Checker

Great Expectations data quality checker for Spark DataFrames.

Uses an ephemeral (in-memory) GE context so no filesystem store is required. This makes it safe to use from Fabric and Databricks notebooks without any additional setup.

exception scamander.quality.checker.DataQualityError

Basisklassen: Exception

Raised when a quality suite fails and raise_on_failure=True.

class scamander.quality.checker.ExpectationResult(expectation_type: str, column: str | list[str] | None, success: bool, unexpected_count: int | None = None, unexpected_percent: float | None = None)

Basisklassen: object

Outcome of a single GE expectation.

class scamander.quality.checker.QualityChecker(spark: SparkSession, log: HumanLogger | None = None)

Basisklassen: object

Validates a Spark DataFrame against a QualitySuiteConfig.

Creates a fresh Great Expectations ephemeral context per check() call so there are no naming conflicts when the same instance is used repeatedly.

Parameters:

  • spark – Active SparkSession.

  • log – Optional HumanLogger (platform.log). Validation summary and per-failure warnings are emitted when a logger is provided.

Example:

checker = QualityChecker(spark=platform.spark, log=platform.log)
result = checker.check(df, suite_config)
if not result.success:
    raise RuntimeError(str(result))
check(df: DataFrame, suite_config: QualitySuiteConfig) → QualityResult

Validate df against all expectations in suite_config.

Parameters:

  • df – Spark DataFrame to validate.

  • suite_config – Suite definition containing expectation specs.

Returns:

QualityResult with per-expectation outcomes.

Veroorzaakt:

  • ImportError – When great-expectations is not installed.

  • DataQualityError – When suite_config.raise_on_failure is True and one or more expectations fail.

class scamander.quality.checker.QualityResult(suite_name: str, success: bool, evaluated_expectations: int, successful_expectations: int, failed_expectations: int, results: list[ExpectationResult] = <factory>)

Basisklassen: object

Aggregated outcome of a full GE expectation suite run.