QualityReporter

Quality run reporter — persists check history and non-compliant rows to Delta tables.

class scamander.quality.reporter.QualityReporter(spark: SparkSession, history_table: str, violations_table: str | None = None, log: HumanLogger | None = None)

Basisklassen: object

Persists QualityResult history and non-compliant rows to Delta tables for longitudinal tracking.

History table (history_table, append) — one row per expectation per run:

run_id            STRING     — UUID assigned to this run
run_timestamp     TIMESTAMP  — UTC time the run was saved
dataset_name      STRING     — caller-supplied label (e.g. "customers_bronze")
suite_name        STRING
suite_success     BOOLEAN    — overall pass/fail of the whole suite
expectation_type  STRING     — e.g. "not_null", "foreign_key"
column            STRING     — target column(s), or null for table-level checks
expectation_success BOOLEAN
unexpected_count  LONG       — number of violating rows (null for schema checks)
unexpected_percent DOUBLE    — violation rate (null where not applicable)

Violations table (violations_table, append, optional) — one row per violating row per expectation that had at least one violation:

run_id            STRING
run_timestamp     TIMESTAMP
dataset_name      STRING
suite_name        STRING
expectation_type  STRING
column            STRING
row_data          STRING     — full violating row serialised as JSON
Parameters:

  • spark – Active SparkSession.

  • history_table – Fully-qualified Delta table name for history (e.g. "dq.quality_history").

  • violations_table – Optional fully-qualified Delta table name for non-compliant rows. When omitted, violation rows are not written.

  • log – Optional HumanLogger.

Example:

from scamander.quality import QualityChecker, QualityReporter

checker  = QualityChecker(spark=platform.spark, log=platform.log)
reporter = QualityReporter(
    spark=platform.spark,
    history_table="dq.quality_history",
    violations_table="dq.quality_violations",
    log=platform.log,
)

result = checker.check(df, suite)
run_id = reporter.save(df, result, suite, dataset_name="customers_bronze")
save(df: DataFrame, result: QualityResult, suite_config: QualitySuiteConfig, dataset_name: str | None = None) → str

Persist result to the history table and optionally the violations table.

Parameters:

  • df – The DataFrame that was validated. Used to extract violating rows when violations_table is configured.

  • result – The QualityResult returned by check().

  • suite_config – The suite that produced result. Expectations must be in the same order as result.results.

  • dataset_name – Human-readable label for the validated dataset stored in both output tables.

Returns:

The run_id (UUID string) assigned to this run.