# Standard ETL Project

## Overview


This project was scaffolded using the **ACC CLI** a developer tool that generates
a standardised Databricks project structure.

By selecting the **Standard** template during `acc init`, the CLI automatically
provisions a layered architecture (Extract → Transform → Load) built on PySpark and
Delta Lake, so you can focus on writing business logic instead of setting up boilerplate.

## Project Structure

```
{{cookiecutter.project_slug}}/
│
├── configs/
│   ├── dev.yml
│   └── prod.yml
│
├── jobs/
│   └── databricks_job.json
│
├── src/
│   ├── task/
        └── __init__.py
│   │   └── extract.py
│   │   └── transform.py
│   │   └── load.py
│   │   └── pipeline.py
│
│   ├── utils/
        └── __init__.py
│   │   └── helpers.py
│   ├── __init__.py
│   ├── main.py
│
├── tests/
│   ├── __init__.py
│   ├── test_extract.py
│   ├── test_transform.py
│   ├── test_load.py
│   ├── test_pipeline.py
│
├── .gitignore
├── __init__.py
├── Makefile
├── pyproject.toml
├── README.md
├── requirements.txt
└── requirements-dev.txt
```

## How It Works

### `configs/dev.yml` & `configs/prod.yml`
Environment-specific configuration files. Hold source paths, target paths,
and format settings for dev and prod environments.

### `jobs/databricks_job.json`
Databricks job definition. Defines the cluster config, task dependencies,
and entry point for running the pipeline on Databricks.

### `src/tasks/extract.py`
Reads raw data from the configured source path and format (parquet, csv, etc.)
using Spark. Returns a raw DataFrame for downstream processing.

### `src/tasks/transform.py`
Applies business logic and data quality rules to the extracted DataFrame —
null handling, deduplication, type casting, and custom transformations.

### `src/tasks/load.py`
Writes the transformed DataFrame to the configured Delta Lake target path
in overwrite mode.

### `src/tasks/pipeline.py`
Orchestrates the full ETL sequence — calls `extract → transform → load` in order
via `run_pipeline(spark, config)`.

### `src/utils/helpers.py`
Shared utility functions used across tasks. Includes helpers for Spark session
creation, config loading, logging setup, and common DataFrame operations.

### `src/main.py`
Main entry point for the pipeline. Loads config, initialises the Spark session,
and triggers `run_pipeline()`.

### `tests/test_extract.py`
Unit tests for `extract.py`. Validates source reading logic and schema expectations.

### `tests/test_transform.py`
Unit tests for `transform.py`. Validates transformation logic and output DataFrame structure.

### `tests/test_load.py`
Unit tests for `load.py`. Validates Delta write behaviour and target path correctness.

### `tests/test_pipeline.py`
Integration tests for `pipeline.py`. Validates the full ETL sequence end-to-end.

### `Makefile`
Convenience commands for common tasks — `make build`, `make test`, `make deploy`.

### `pyproject.toml`
Project metadata and dependency management. Defines the package name, version,
and build system configuration.

#### Optional Cloud & Databricks Dependencies
If your pipeline needs to upload or interact with Amazon S3, Azure Data Lake Storage Gen2, Azure Blob Storage, or Databricks Unity Catalog, the required dependencies can be added directly to the generated pyproject.toml file.

### `requirements.txt` & `requirements-dev.txt`
Runtime and development dependencies respectively. `requirements-dev.txt` includes
testing and linting libraries on top of the base requirements.

### Building a `.whl` File
To package this project as a Python wheel (`.whl`) for deployment to Databricks, AWS or Azure,
run the following command from the project root:

```bash
acc build
```

This will compile the project into a distributable `.whl` file