# Medallion Architecture Project

## Overview

This project was scaffolded using the **ACC CLI** a developer tool that generates
a standardised Databricks project structure.

By selecting the **Medallion** template during `acc init`, the CLI automatically
provisions a layered architecture (Bronze → Silver → Gold) built on PySpark and
Delta Lake, so you can focus on writing business logic instead of setting up boilerplate.

## Project Structure

```
{{cookiecutter.project_slug}}/
│
├── configs/
│   ├── dev.yml
│   └── prod.yml
│
├── jobs/
│   └── databricks_job.json
│
├── src/
│   ├── bronze/
        └── __init__.py
│   │   └── ingest.py
│   ├── silver/
        └── __init__.py
│   │   └── cleanse.py
│   ├── gold/
        └── __init__.py
│   │   └── marts.py
│   ├── utils/
        └── __init__.py
│   │   └── helpers.py
│   ├── __init__.py
│   ├── main.py
│
├── tests/
│   ├── __init__.py
│   ├── test_bronze.py
│   ├── test_silver.py
│   ├── test_gold.py
│
├── .gitignore
├── __init__.py
├── Makefile
├── pyproject.toml
├── README.md
├── requirements.txt
└── requirements-dev.txt
```
## How It Works

### `configs/dev.yml` & `configs/prod.yml`
Environment-specific configuration files. Hold source paths, target paths, and
layer-specific settings (bronze_path, silver_path, gold_path) for dev and prod environments.

### `jobs/databricks_job.json`
Databricks job definition. Defines the cluster config, task dependencies,
and entry point for running the pipeline on Databricks.

### `src/bronze/ingest.py`
Entry point for the Bronze layer. Reads raw data as-is from the source
(parquet, csv, etc.) and writes it to the Bronze table without any transformations.

### `src/silver/cleanse.py`
Processes Bronze data into the Silver layer. Applies data quality rules,
drops nulls, removes duplicates, standardizes schema and writes to the Silver Delta table.

### `src/gold/marts.py`
Aggregates Silver data into the Gold layer. Builds business-ready data marts
with grouped metrics and KPIs, writing the final output to the Gold Delta table.

### `src/utils/helpers.py`
Shared utility functions used across layers. Includes helpers for Spark session
creation, config loading, logging setup, and common DataFrame operations.

### `src/main.py`
Main entry point for the pipeline. Loads config, initialises the Spark session,
and orchestrates the Bronze → Silver → Gold execution sequence.

### `tests/test_bronze.py`
Unit tests for `ingest.py`. Validates raw ingestion logic, schema expectations,
and Bronze Delta write behaviour.

### `tests/test_silver.py`
Unit tests for `cleanse.py`. Validates null handling, deduplication logic,
and Silver layer output schema.

### `tests/test_gold.py`
Unit tests for `marts.py`. Validates aggregation logic and Gold mart output
against expected results.

### `Makefile`
Convenience commands for common tasks — `make build`, `make test`, `make deploy`.

### `pyproject.toml`
Project metadata and dependency management. Defines the package name, version,
and build system configuration.

#### Optional Cloud & Databricks Dependencies
If your pipeline needs to upload or interact with Amazon S3, Azure Data Lake Storage Gen2, Azure Blob Storage, or Databricks Unity Catalog, the required dependencies can be added directly to the generated pyproject.toml file.

### `requirements.txt` & `requirements-dev.txt`
Runtime and development dependencies respectively. `requirements-dev.txt` includes
testing and linting libraries on top of the base requirements.


### Building a `.whl` File
To package this project as a Python wheel (`.whl`) for deployment to Databricks, AWS or Azure,
run the following command from the project root:

```bash
acc build
```

This will compile the project into a distributable `.whl` file