# Full Pipeline Project

## Overview

This project was scaffolded using the ACC CLI — a developer tool that generates a standardised Databricks project structure.

By selecting the **Full Pipeline** template during `acc init`, the CLI automatically provisions a combined ML architecture (Data Loading → Validation → Train → Save Model → Load Model → Validate Input → Predict → Output) built on PySpark and MLflow, so you can run end-to-end training and inference from a single project without managing two separate codebases.

## Project Structure
```

{{cookiecutter.project_slug}}/
│
├── configs/
│   ├── dev.yml
│   ├── prod.yml
│   ├── training_config.yml
│   └── inference_config.yml
│
├── jobs/
│   ├── training_job.json
│   ├── inference_job.json
│   └── full_pipeline_job.json
│
├── src/
│   ├── training/
│   │   ├── __init__.py
│   │   ├── data_loader.py
│   │   ├── validator.py
│   │   ├── trainer.py
│   │   └── model_saver.py
│   ├── inference/
│   │   ├── __init__.py
│   │   ├── model_loader.py
│   │   ├── input_validator.py
│   │   ├── predictor.py
│   │   └── output_handler.py
│   ├── utils/
│   │   ├── __init__.py
│   │   └── helpers.py
│   ├── __init__.py
│   └── main.py
│
├── tests/
│   ├── __init__.py
│   ├── training/
│   │   ├── __init__.py
│   │   ├── test_data_loader.py
│   │   ├── test_validator.py
│   │   ├── test_trainer.py
│   │   └── test_model_saver.py
│   └── inference/
│       ├── __init__.py
│       ├── test_model_loader.py
│       ├── test_input_validator.py
│       ├── test_predictor.py
│       └── test_output_handler.py
│
├── .gitignore
├── __init__.py
├── Makefile
├── pyproject.toml
├── README.md
├── requirements.txt
└── requirements-dev.txt
```


## How It Works

### `configs/dev.yml` & `configs/prod.yml`
Environment-specific configuration files. Hold shared settings applicable to both training and inference such as Spark configs, MLflow URIs, and workspace paths.

### `configs/training_config.yml` & `configs/inference_config.yml`
Pipeline-specific configuration files. `training_config.yml` holds data source paths, feature settings, and model hyperparameters. `inference_config.yml` holds model registry references, input sources, and output target paths.

### `jobs/training_job.json` & `jobs/inference_job.json`
Individual Databricks job definitions for running training or inference independently. Useful for scheduling retraining jobs separately from batch scoring jobs.

### `jobs/full_pipeline_job.json`
Combined Databricks job definition that chains training and inference as dependent tasks in a single job run. Defines task order, cluster configs, and shared parameters.

### `src/training/data_loader.py`
Reads raw training data from the configured source and returns a DataFrame ready for validation.

### `src/training/validator.py`
Validates the training dataset. Checks schema, nulls, distributions, and feature completeness before training begins.

### `src/training/trainer.py`
Fits the model on validated training data. Logs parameters, metrics, and artefacts to MLflow for experiment tracking.

### `src/training/model_saver.py`
Saves the trained model to the MLflow Model Registry or Unity Catalog with versioning and stage transition support.

### `src/inference/model_loader.py`
Loads the registered model from the MLflow Model Registry by name and version for use in prediction.

### `src/inference/input_validator.py`
Validates incoming inference data against the model's expected feature schema before predictions are made.

### `src/inference/predictor.py`
Applies the loaded model to validated input data and generates predictions with scores or confidence metrics.

### `src/inference/output_handler.py`
Formats and writes prediction results to the configured output target with schema enforcement and run metadata.

### `src/utils/helpers.py`
Shared utility functions used across both pipelines. Includes helpers for Spark session creation, config loading, MLflow setup, logging, and common DataFrame operations.

### `src/main.py`
Main entry point for the full pipeline. Loads config, initialises the Spark session, and orchestrates the complete Training → Inference execution sequence end to end.

### `tests/training/` & `tests/inference/`
Mirrored test directories for each pipeline. Each step file has a corresponding test file validating its logic, schema expectations, and output behaviour in isolation.

### `Makefile`
Convenience commands for common tasks — `make build`, `make test`, `make lint`, `make format`.

### `pyproject.toml`
Project metadata and dependency management. Defines the package name, version, and build system configuration.

#### Optional Cloud & Databricks Dependencies
If your pipeline needs to upload or interact with Amazon S3, Azure Data Lake Storage Gen2, Azure Blob Storage, or Databricks Unity Catalog, the required dependencies can be added directly to the generated pyproject.toml file.


### `requirements.txt` & `requirements-dev.txt`
Runtime and development dependencies respectively. `requirements-dev.txt` includes testing and linting libraries on top of the base requirements.

## Building a `.whl` File

To package this project as a Python wheel (`.whl`) for deployment to Databricks, AWS or Azure, run the following command from the project root:

\`\`\`
acc build
\`\`\`

This will compile the project into a distributable `.whl` file.