API reference ============== .. include:: _autodoc/mdif.inc.rst The generated reference above (when ``Scamander-MDIF`` is installed in the build environment) is the authoritative, always-current source of docstrings, method signatures, and parameters, organised by package. The summaries below are a hand-maintained, high-level overview of what each part is for — for exact signatures, follow the links above. Platform -------- ``scamander.processor.platform.Platform`` is the entry point for every MDIF pipeline: instantiate it and it auto-detects the runtime and returns the correct subclass (``Databricks`` or ``Fabric``). It wraps the platform-specific parts of ingestion — reading config files, checking and moving files, creating and writing to Delta tables (including upserts and SCD Type-2 history), and structured logging — behind one consistent interface. Databricks and Fabric ---------------------- ``Databricks`` and ``Fabric`` both extend ``Platform`` with the bits that differ per runtime. ``Databricks`` adds access to the Databricks Utilities object and full ``PRIMARY KEY``/``FOREIGN KEY`` constraint support when creating tables. ``Fabric`` adds Lakehouse discovery and compatibility checks, and translates OneLake paths to local mount paths. Source connectors ------------------ Every connector — CSV, JSON, Oracle, PostgreSQL, SQL Server, Lakehouse — shares one method: ``df = connector.connect()``. Each reads its source using Spark's native readers where possible, or JDBC for the database sources. See *Supported source types* below for which ``config.source.type`` string selects which connector. Transformer ----------- ``scamander.transformations.Transformer`` is a fluent, chainable builder covering the full bronze→silver→gold path: type casting, cleaning (trimming, renaming, deduplication), joins, aggregations, derived columns, and pivots. Every method returns ``self`` for chaining except ``validate()``, which is terminal. It also plugs into anonymization (``.anonymize(config)``) and quality validation (``.validate(config)``). The underlying cleaning/enrichment/SCD functions are also available as standalone pure functions for use outside the builder. QualityChecker and QualityReporter ------------------------------------ ``QualityChecker`` runs Great Expectations suites against a Spark DataFrame using an ephemeral in-memory context — no GE project or filesystem store needed — and can raise on failure. ``QualityReporter`` persists a run's history, and any violating rows, to Delta tables for later inspection. See the expectation types table in :doc:`usage-guide` for the config side of quality suites. Anonymizer ---------- ``scamander.anonymization.Anonymizer`` applies a declarative set of anonymization rules — hash, mask, redact, generalize, encrypt, fake — to a DataFrame, driven by an ``AnonymizationConfig``. It plugs directly into the ``Transformer`` chain via ``.anonymize(config)``. Utils ----- ``scamander.utils`` holds small, stdlib-only filesystem helpers: safely extracting archives, and recursively reading a directory of JSON files into typed records. Loggers ------- ``HumanLogger`` writes to stdout for interactive/console use; ``MachineLogger`` writes structured log rows to a Delta table for later querying. Both share the same level methods (``debug``, ``info``, ``warning``, ``error``, ``critical``, ``exception``) and are session- and calling-function-aware. OAuth2Session ------------- ``scamander.api.oauth2_session.OAuth2Session`` extends ``requests.Session`` with automatic OAuth 2.0 client-credentials token refresh, so REST API calls made through it never need to manage tokens by hand. Secret providers ----------------- ``scamander.secrets`` abstracts secret retrieval behind one interface — pass any provider to ``Platform(secret_provider=...)``. Backends are provided for HashiCorp Vault, Azure Key Vault, and Databricks secret scopes. Supported source types ------------------------ .. list-table:: :header-rows: 1 * - ``config.source.type`` - Connector class - Status * - ``"csv"`` - ``CsvConnector`` - Stable * - ``"json"`` - ``JsonConnector`` - Stable * - ``"oracle"`` - ``OracleConnector`` - Stable * - ``"postgresql"`` - ``PostgresConnector`` - Stable * - ``"sql"`` - ``SqlServerConnector`` - In progress * - ``"LakeHouse"`` - ``LakeHouseConnector`` - In progress Datatype mapping ------------------ Source system type strings in metadata configs are normalised to Delta Lake types by ``Platform.map_datatype``. The mapping is intentionally broad so config files can use native source-system type names directly, including Oracle, SQL Server, and PostgreSQL types. .. list-table:: :header-rows: 1 * - Source types accepted - Delta / Spark SQL type * - ``VARCHAR``, ``NVARCHAR``, ``CHAR``, ``NCHAR``, ``TEXT``, ``NTEXT``, ``STRING``, ``UNIQUEIDENTIFIER``, ``UUID``, ``JSON``, ``XML`` - ``STRING`` * - ``VARCHAR2``, ``NVARCHAR2``, ``CLOB``, ``NCLOB``, ``LONG`` (Oracle) - ``STRING`` * - ``XMLTYPE``, ``ROWID``, ``UROWID``, ``INTERVAL YEAR TO MONTH``, ``INTERVAL DAY TO SECOND`` (Oracle) - ``STRING`` * - ``BOOLEAN``, ``BOOL``, ``BIT`` - ``BOOLEAN`` * - ``TINYINT``, ``BYTE`` - ``TINYINT`` * - ``SMALLINT`` - ``SMALLINT`` * - ``INT``, ``INTEGER`` - ``INT`` * - ``BIGINT`` - ``BIGINT`` * - ``REAL``, ``FLOAT``, ``BINARY_FLOAT`` (Oracle) - ``FLOAT`` * - ``DOUBLE``, ``FLOAT8``, ``BINARY_DOUBLE`` (Oracle) - ``DOUBLE`` * - ``NUMERIC``, ``DECIMAL``, ``MONEY``, ``SMALLMONEY``, ``NUMBER`` (Oracle) - ``DECIMAL(precision, scale)`` * - ``DATE`` - ``DATE`` * - ``DATETIME``, ``DATETIME2``, ``SMALLDATETIME``, ``TIMESTAMP``, ``TIME``, ``DATETIMEOFFSET`` - ``TIMESTAMP`` * - ``TIMESTAMP WITH TIME ZONE``, ``TIMESTAMP WITH LOCAL TIME ZONE`` (Oracle) - ``TIMESTAMP`` * - ``BINARY``, ``VARBINARY``, ``IMAGE`` - ``BINARY`` * - ``RAW``, ``LONG RAW``, ``BLOB``, ``BFILE`` (Oracle) - ``BINARY`` * - Anything unrecognised - ``STRING`` For ``DECIMAL``/``NUMBER`` columns, ``precision`` and ``scale`` are read from the column metadata dict and default to ``(10, 0)`` when not supplied. SCD Type-2 history tracking ------------------------------ Setting ``"track_history": true`` in a metadata config causes ``create_empty_table_from_metadata`` to append five SCD Type-2 history columns to the generated DDL automatically: .. list-table:: :header-rows: 1 * - Column - Delta type - Description * - ``HashSleutel`` - ``STRING NOT NULL`` - SHA-256 hash of the natural key columns — uniquely identifies an entity version * - ``HashWaarde`` - ``STRING NOT NULL`` - SHA-256 hash of all tracked attribute columns — changes when any attribute changes * - ``GeldigVan`` - ``TIMESTAMP NOT NULL`` - Start of the validity period for this record version * - ``GeldigTot`` - ``TIMESTAMP`` - End of the validity period; ``NULL`` means the record is currently active * - ``IsActief`` - ``BOOLEAN NOT NULL`` - ``true`` for the current, active version of each entity Use ``apply_scd2_merge`` (see :doc:`usage-guide`) to run the merge automatically. For a manual merge: compute ``HashSleutel`` from key columns and look up the active row (``IsActief = true``); compute ``HashWaarde`` from attribute columns and compare to the stored hash; on change, set ``GeldigTot = now()``/``IsActief = false`` on the old row and insert a new row with ``GeldigVan = now()``/``IsActief = true``.