Suite

Pydantic models for data quality suite configuration.

A QualitySuiteConfig maps directly onto a Great Expectations ExpectationSuite and can be loaded from the same JSON/YAML config files used by the rest of the framework.

class scamander.models.quality_config.suite.ExpectationSpec(*, type: Literal['not_null', 'unique', 'multi_column_unique', 'between', 'in_set', 'not_in_set', 'regex', 'row_count_between', 'column_exists', 'column_type', 'foreign_key', 'column_comparison', 'conditional_not_null', 'custom_sql'], column: str | None = None, columns: list[str] | None = None, min_value: float | None = None, max_value: float | None = None, value_set: list[Any] | None = None, regex: str | None = None, type_: str | None = None, mostly: float = 1.0, ref_table: str | None = None, ref_column: str | None = None, expression: str | None = None, when_column: str | None = None, then_column: str | None = None, sql: str | None = None)

Bases: BaseModel

Configuration for a single data quality expectation.

Most types delegate to a native Great Expectations expectation. Four types use GE’s UnexpectedRowsExpectation to run Spark-SQL queries — they do not support the mostly parameter:

• foreign_key — every non-null value in column must exist in ref_column of ref_table (a registered Spark table or temp view).

• column_comparison — the SQL expression must evaluate to true for every row (e.g. "start_date <= end_date").

• conditional_not_null — when when_column is not null, then_column must also be not null.

• custom_sql — the sql query must return zero rows to pass. Use {batch} in the query to reference the DataFrame being validated.

type

Expectation type identifier.

Type:

ExpectationType

column

Target column (required for most column-level expectations).

Type:

str | None

columns

List of columns for multi_column_unique.

Type:

list[str] | None

min_value

Lower bound for between and row_count_between.

Type:

float | None

max_value

Upper bound for between and row_count_between.

Type:

float | None

value_set

Allowed (or disallowed) values for in_set / not_in_set.

Type:

list[Any] | None

regex

Regular expression pattern for regex.

Type:

str | None

type_

Expected Spark type string for column_type (e.g. "StringType").

Type:

str | None

mostly

Fraction of rows that must satisfy the expectation (0.0–1.0). Defaults to 1.0 (all rows). Not applicable to row_count_between, column_exists, multi_column_unique, foreign_key, column_comparison, conditional_not_null, or custom_sql.

Type:

float

ref_table

Reference table or temp-view name for foreign_key.

Type:

str | None

ref_column

Column in ref_table whose values form the allowed set.

Type:

str | None

expression

SQL boolean expression for column_comparison (e.g. "start_date <= end_date").

Type:

str | None

when_column

Trigger column for conditional_not_null.

Type:

str | None

then_column

Required column for conditional_not_null.

Type:

str | None

sql

Full Spark-SQL query for custom_sql. Must return zero rows to pass. Use {batch} to reference the DataFrame being validated.

Type:

str | None

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class scamander.models.quality_config.suite.QualitySuiteConfig(*, name: str, raise_on_failure: bool = False, expectations: list[ExpectationSpec])

Bases: BaseModel

Top-level data quality suite configuration.

name

Suite name — also used as the GE ExpectationSuite name.

Type:

str

raise_on_failure

When True, a failed validation raises DataQualityError instead of returning the result silently.

Type:

bool

expectations

Ordered list of expectation specifications to run.

Type:

list[ExpectationSpec]

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].