API reference¶

The generated reference above (when Scamander-MDIF is installed in the build environment) is the authoritative, always-current source of docstrings, method signatures, and parameters, organised by package. The summaries below are a hand-maintained, high-level overview of what each part is for — for exact signatures, follow the links above.

Platform¶

scamander.processor.platform.Platform is the entry point for every MDIF pipeline: instantiate it and it auto-detects the runtime and returns the correct subclass (Databricks or Fabric). It wraps the platform-specific parts of ingestion — reading config files, checking and moving files, creating and writing to Delta tables (including upserts and SCD Type-2 history), and structured logging — behind one consistent interface.

Databricks and Fabric¶

Databricks and Fabric both extend Platform with the bits that differ per runtime. Databricks adds access to the Databricks Utilities object and full PRIMARY KEY/FOREIGN KEY constraint support when creating tables. Fabric adds Lakehouse discovery and compatibility checks, and translates OneLake paths to local mount paths.

Source connectors¶

Every connector — CSV, JSON, Oracle, PostgreSQL, SQL Server, Lakehouse — shares one method: df = connector.connect(). Each reads its source using Spark’s native readers where possible, or JDBC for the database sources. See Supported source types below for which config.source.type string selects which connector.

Transformer¶

scamander.transformations.Transformer is a fluent, chainable builder covering the full bronze→silver→gold path: type casting, cleaning (trimming, renaming, deduplication), joins, aggregations, derived columns, and pivots. Every method returns self for chaining except validate(), which is terminal. It also plugs into anonymization (.anonymize(config)) and quality validation (.validate(config)). The underlying cleaning/enrichment/SCD functions are also available as standalone pure functions for use outside the builder.

QualityChecker and QualityReporter¶

QualityChecker runs Great Expectations suites against a Spark DataFrame using an ephemeral in-memory context — no GE project or filesystem store needed — and can raise on failure. QualityReporter persists a run’s history, and any violating rows, to Delta tables for later inspection. See the expectation types table in Usage guide for the config side of quality suites.

Anonymizer¶

scamander.anonymization.Anonymizer applies a declarative set of anonymization rules — hash, mask, redact, generalize, encrypt, fake — to a DataFrame, driven by an AnonymizationConfig. It plugs directly into the Transformer chain via .anonymize(config).

Utils¶

scamander.utils holds small, stdlib-only filesystem helpers: safely extracting archives, and recursively reading a directory of JSON files into typed records.

Loggers¶

HumanLogger writes to stdout for interactive/console use; MachineLogger writes structured log rows to a Delta table for later querying. Both share the same level methods (debug, info, warning, error, critical, exception) and are session- and calling-function-aware.

OAuth2Session¶

scamander.api.oauth2_session.OAuth2Session extends requests.Session with automatic OAuth 2.0 client-credentials token refresh, so REST API calls made through it never need to manage tokens by hand.

Secret providers¶

scamander.secrets abstracts secret retrieval behind one interface — pass any provider to Platform(secret_provider=...). Backends are provided for HashiCorp Vault, Azure Key Vault, and Databricks secret scopes.

Supported source types¶

`config.source.type`	Connector class	Status
`"csv"`	`CsvConnector`	Stable
`"json"`	`JsonConnector`	Stable
`"oracle"`	`OracleConnector`	Stable
`"postgresql"`	`PostgresConnector`	Stable
`"sql"`	`SqlServerConnector`	In progress
`"LakeHouse"`	`LakeHouseConnector`	In progress

Datatype mapping¶

Source system type strings in metadata configs are normalised to Delta Lake types by Platform.map_datatype. The mapping is intentionally broad so config files can use native source-system type names directly, including Oracle, SQL Server, and PostgreSQL types.

Source types accepted	Delta / Spark SQL type
`VARCHAR`, `NVARCHAR`, `CHAR`, `NCHAR`, `TEXT`, `NTEXT`, `STRING`, `UNIQUEIDENTIFIER`, `UUID`, `JSON`, `XML`	`STRING`
`VARCHAR2`, `NVARCHAR2`, `CLOB`, `NCLOB`, `LONG` (Oracle)	`STRING`
`XMLTYPE`, `ROWID`, `UROWID`, `INTERVAL YEAR TO MONTH`, `INTERVAL DAY TO SECOND` (Oracle)	`STRING`
`BOOLEAN`, `BOOL`, `BIT`	`BOOLEAN`
`TINYINT`, `BYTE`	`TINYINT`
`SMALLINT`	`SMALLINT`
`INT`, `INTEGER`	`INT`
`BIGINT`	`BIGINT`
`REAL`, `FLOAT`, `BINARY_FLOAT` (Oracle)	`FLOAT`
`DOUBLE`, `FLOAT8`, `BINARY_DOUBLE` (Oracle)	`DOUBLE`
`NUMERIC`, `DECIMAL`, `MONEY`, `SMALLMONEY`, `NUMBER` (Oracle)	`DECIMAL(precision, scale)`
`DATE`	`DATE`
`DATETIME`, `DATETIME2`, `SMALLDATETIME`, `TIMESTAMP`, `TIME`, `DATETIMEOFFSET`	`TIMESTAMP`
`TIMESTAMP WITH TIME ZONE`, `TIMESTAMP WITH LOCAL TIME ZONE` (Oracle)	`TIMESTAMP`
`BINARY`, `VARBINARY`, `IMAGE`	`BINARY`
`RAW`, `LONG RAW`, `BLOB`, `BFILE` (Oracle)	`BINARY`
Anything unrecognised	`STRING`

For DECIMAL/NUMBER columns, precision and scale are read from the column metadata dict and default to (10, 0) when not supplied.

SCD Type-2 history tracking¶

Setting "track_history": true in a metadata config causes create_empty_table_from_metadata to append five SCD Type-2 history columns to the generated DDL automatically:

Column	Delta type	Description
`HashSleutel`	`STRING NOT NULL`	SHA-256 hash of the natural key columns — uniquely identifies an entity version
`HashWaarde`	`STRING NOT NULL`	SHA-256 hash of all tracked attribute columns — changes when any attribute changes
`GeldigVan`	`TIMESTAMP NOT NULL`	Start of the validity period for this record version
`GeldigTot`	`TIMESTAMP`	End of the validity period; `NULL` means the record is currently active
`IsActief`	`BOOLEAN NOT NULL`	`true` for the current, active version of each entity

Use apply_scd2_merge (see Usage guide) to run the merge automatically. For a manual merge: compute HashSleutel from key columns and look up the active row (IsActief = true); compute HashWaarde from attribute columns and compare to the stored hash; on change, set GeldigTot = now()/IsActief = false on the old row and insert a new row with GeldigVan = now()/IsActief = true.