# Inference Pipeline Project

## Overview

This project was scaffolded using the ACC CLI — a developer tool that generates a standardised Databricks project structure.

By selecting the **Inference** template during `acc init`, the CLI automatically provisions a step-based ML inference architecture (Load Model → Validate Input → Predict → Output) built on PySpark and MLflow, so you can focus on writing prediction logic instead of setting up boilerplate.

## Project Structure
```

{{cookiecutter.project_slug}}/
│
├── configs/
│   ├── dev.yml
│   └── prod.yml
│
├── jobs/
│   └── inference_job.json
│
├── src/
│   ├── inference/
│   │   ├── __init__.py
│   │   ├── model_loader.py
│   │   ├── input_validator.py
│   │   ├── predictor.py
│   │   └── output_handler.py
│   ├── utils/
│   │   ├── __init__.py
│   │   └── helpers.py
│   ├── __init__.py
│   └── main.py
│
├── tests/
│   ├── __init__.py
│   ├── test_model_loader.py
│   ├── test_input_validator.py
│   ├── test_predictor.py
│   └── test_output_handler.py
│
├── .gitignore
├── __init__.py
├── Makefile
├── pyproject.toml
├── README.md
├── requirements.txt
└── requirements-dev.txt
```


## How It Works

### `configs/dev.yml` & `configs/prod.yml`
Environment-specific configuration files. Hold model registry paths, input data sources, output target paths, and inference-specific settings for dev and prod environments.

### `jobs/inference_job.json`
Databricks job definition. Defines the cluster config, task dependencies, and entry point for running the inference pipeline on Databricks.

### `src/inference/model_loader.py`
Entry point for the inference pipeline. Loads the registered model from the MLflow Model Registry or Unity Catalog by name and version, and returns it ready for prediction.

### `src/inference/input_validator.py`
Validates incoming data before predictions are made. Checks schema alignment with the model's expected feature set, null thresholds, and data type correctness. Raises early if input does not meet requirements.

### `src/inference/predictor.py`
Core prediction logic. Applies the loaded model to the validated input DataFrame, generates predictions, and attaches probability scores or confidence metrics where applicable.

### `src/inference/output_handler.py`
Formats and writes prediction results to the configured output target (Delta table, parquet, etc.). Handles column renaming, schema enforcement, and appending metadata such as run timestamps and model version.

### `src/utils/helpers.py`
Shared utility functions used across the pipeline. Includes helpers for Spark session creation, config loading, MLflow setup, logging, and common DataFrame operations.

### `src/main.py`
Main entry point for the pipeline. Loads config, initialises the Spark session, and orchestrates the Load Model → Validate Input → Predict → Output execution sequence.

### `tests/test_model_loader.py`
Unit tests for `model_loader.py`. Validates model loading logic, registry path resolution, and that the returned model object meets expected interface requirements.

### `tests/test_input_validator.py`
Unit tests for `input_validator.py`. Validates schema checks, null handling, and that invalid input DataFrames are correctly caught and rejected before prediction.

### `tests/test_predictor.py`
Unit tests for `predictor.py`. Validates prediction logic, output column structure, and scoring behaviour against known inputs and expected outputs.

### `tests/test_output_handler.py`
Unit tests for `output_handler.py`. Validates output formatting, Delta write behaviour, schema enforcement, and metadata attachment logic.

### `Makefile`
Convenience commands for common tasks — `make build`, `make test`, `make lint`, `make format`.

### `pyproject.toml`
Project metadata and dependency management. Defines the package name, version, and build system configuration.

#### Optional Cloud & Databricks Dependencies
If your pipeline needs to upload or interact with Amazon S3, Azure Data Lake Storage Gen2, Azure Blob Storage, or Databricks Unity Catalog, the required dependencies can be added directly to the generated pyproject.toml file.


### `requirements.txt` & `requirements-dev.txt`
Runtime and development dependencies respectively. `requirements-dev.txt` includes testing and linting libraries on top of the base requirements.

## Building a `.whl` File

To package this project as a Python wheel (`.whl`) for deployment to Databricks, AWS or Azure, run the following command from the project root:

\`\`\`
acc build
\`\`\`

This will compile the project into a distributable `.whl` file.