Scamander MDIF

Scamander (Metadata-Driven Ingestion Framework, or MDIF) is a multi-platform Python SDK for metadata-driven data ingestion and transformation on Spark-based platforms. It abstracts away the differences between Microsoft Fabric and Databricks, so one ingestion or transformation pipeline runs unchanged on either, or on vanilla PySpark for local testing.

The platform’s Airflow providers (see Custom Airflow providers) schedule when and where a job runs. MDIF is the library that job’s own code imports: the notebook kernel a SparkNotebookOperator task runs is a Spark driver, and it’s usually MDIF doing the actual reading, transforming, and writing inside it.

A few things MDIF gives you out of the box:

  • The same Python code runs on Fabric, Databricks, and local Spark with no changes.

  • Delta tables get created straight from JSON metadata, including data types, nullability, primary/foreign keys, and SCD Type-2 history columns.

  • A chainable Transformer API covers the full bronze→silver→gold path: type casting, cleaning, joins, aggregations, window functions, and pivots.

  • QualityChecker runs Great Expectations suites against any Spark DataFrame from a config-driven expectation model, logging results and optionally raising on failure.

  • Anonymizer handles anonymization and pseudonymization: six techniques (hash, mask, redact, generalize, encrypt, fake) driven by a declarative AnonymizationConfig, and it plugs straight into the Transformer chain as .anonymize(config).

  • Every pipeline run produces human-readable console logs, which are automatically forwarded into structured machine logs persisted to a Delta table.

  • CSV, JSON, Oracle, SQL Server, PostgreSQL, and Lakehouse sources are all accessed through the same Source.connect() interface.

  • A session wrapper takes care of OAuth 2.0 token refresh for REST API calls, so you don’t have to.

Architecture

src/scamander/
├── processor/          # Platform (abstract base, auto-detecting factory),
│                       # Fabric, Databricks, Spark (vanilla fallback)
├── sources/             # Source (factory) + csv/json/oracle/postgresql/
│                       # sqlserver/lakehouse connectors
├── sinks/               # Sink (factory) + lakehouse_sink
├── transformations/    # Transformer (fluent builder), cleaning
│                       # (bronze→silver), enrichment (silver→gold), scd
├── quality/             # QualityChecker, QualityReporter
├── anonymization/      # Anonymizer + pure anonymization functions
├── secrets/             # SecretProvider + Vault / Azure Key Vault /
│                       # Databricks implementations
├── models/              # Pydantic config models: Config, per-source and
│                       # per-sink options, transformation/quality/
│                       # anonymization config
├── utils/               # extract_archive, read_json_records
├── logging/             # HumanLogger (stdout), MachineLogger (Delta table)
└── api/                 # OAuth2Session (auto-refreshing requests.Session)

Contents