Scamander MDIF
================

**Scamander** (Metadata-Driven Ingestion Framework, or MDIF) is a
multi-platform Python SDK for metadata-driven data ingestion and
transformation on Spark-based platforms. It abstracts away the
differences between **Microsoft Fabric** and **Databricks**, so one
ingestion or transformation pipeline runs unchanged on either, or on
vanilla PySpark for local testing.

The platform's Airflow providers (see :doc:`../airflow/providers/index`)
schedule *when* and *where* a job runs. MDIF is the library that job's
own code imports: the notebook kernel a ``SparkNotebookOperator`` task
runs *is* a Spark driver, and it's usually MDIF doing the actual
reading, transforming, and writing inside it.

A few things MDIF gives you out of the box:

- The same Python code runs on Fabric, Databricks, and local Spark with
  no changes.
- Delta tables get created straight from JSON metadata, including data
  types, nullability, primary/foreign keys, and SCD Type-2 history
  columns.
- A chainable ``Transformer`` API covers the full bronze→silver→gold
  path: type casting, cleaning, joins, aggregations, window functions,
  and pivots.
- ``QualityChecker`` runs Great Expectations suites against any Spark
  DataFrame from a config-driven expectation model, logging results and
  optionally raising on failure.
- ``Anonymizer`` handles anonymization and pseudonymization: six
  techniques (hash, mask, redact, generalize, encrypt, fake) driven by a
  declarative ``AnonymizationConfig``, and it plugs straight into the
  ``Transformer`` chain as ``.anonymize(config)``.
- Every pipeline run produces human-readable console logs, which are
  automatically forwarded into structured machine logs persisted to a
  Delta table.
- CSV, JSON, Oracle, SQL Server, PostgreSQL, and Lakehouse sources are
  all accessed through the same ``Source.connect()`` interface.
- A session wrapper takes care of OAuth 2.0 token refresh for REST API
  calls, so you don't have to.

Architecture
------------

.. code-block:: text

   src/scamander/
   ├── processor/          # Platform (abstract base, auto-detecting factory),
   │                       # Fabric, Databricks, Spark (vanilla fallback)
   ├── sources/             # Source (factory) + csv/json/oracle/postgresql/
   │                       # sqlserver/lakehouse connectors
   ├── sinks/               # Sink (factory) + lakehouse_sink
   ├── transformations/    # Transformer (fluent builder), cleaning
   │                       # (bronze→silver), enrichment (silver→gold), scd
   ├── quality/             # QualityChecker, QualityReporter
   ├── anonymization/      # Anonymizer + pure anonymization functions
   ├── secrets/             # SecretProvider + Vault / Azure Key Vault /
   │                       # Databricks implementations
   ├── models/              # Pydantic config models: Config, per-source and
   │                       # per-sink options, transformation/quality/
   │                       # anonymization config
   ├── utils/               # extract_archive, read_json_records
   ├── logging/             # HumanLogger (stdout), MachineLogger (Delta table)
   └── api/                 # OAuth2Session (auto-refreshing requests.Session)

.. toctree::
   :maxdepth: 2
   :caption: Contents

   getting-started
   installation
   core-concepts
   usage-guide
   api-reference