Spark¶

Vanilla PySpark platform implementation (local development / CI).

class scamander.processor.spark.Spark(*args, **kwargs)¶

Bases: Platform

Vanilla PySpark platform implementation for local development and CI environments.

Instantiated automatically by Platform.__new__ when neither Fabric nor Databricks environment variables are detected. Uses Python’s standard pathlib / shutil for filesystem operations, which works against any locally-mounted filesystem or object storage path that the OS can resolve.

Example:

from scamander import Platform

# Local dev / CI: no Fabric or Databricks env vars present, so
# Platform() auto-detects and returns a Spark instance transparently.
platform = Platform()
platform.spark.sql("SELECT 1").show()

check_file_exists(file_name: str, raise_exceptions=False) → bool¶

Checks whether a file or directory exists on the local / mounted filesystem.

Parameters:

file_name (str) – Absolute path to the file or directory.
raise_exceptions (bool) – When True, raises FileNotFoundError if the path does not exist. Defaults to False.

Returns:

True if the path exists, False otherwise.

Return type:

bool

Raises:

FileNotFoundError – Only when raise_exceptions=True and the path is missing.

create_empty_table_from_metadata(metadata: dict)¶

Creates an empty Delta table from a metadata definition dictionary.

Vanilla Spark does not support PRIMARY KEY or FOREIGN KEY constraints in Delta DDL, so only column definitions and NOT NULL constraints are generated. When metadata["track_history"] is True, the five SCD Type-2 columns are appended automatically.

Parameters:: metadata (dict) – Metadata dictionary with "table", "columns", and "track_history" keys.

get_notebook_name() → str¶

Returns a placeholder notebook name for local Spark environments where no notebook context is available.

Returns:: The string "local" as a placeholder.
Return type:: str

get_notebook_url() → str¶

Returns a placeholder URL for local Spark environments where no notebook context is available.

Returns:: The string "local" as a placeholder.
Return type:: str

move_file(source: str, destination: str, overwrite=True, create_missing_dirs: bool = False) → bool¶

Moves a file from source to destination using shutil.move.

Parameters:

source (str) – Absolute path to the source file.
destination (str) – Absolute destination path.
overwrite (bool) – If True (default), remove an existing destination file before moving.
create_missing_dirs (bool) – If True, create the destination directory when it does not exist. Defaults to False.

Returns:

True on success, False on any failure.

Return type:

bool

resolve_filemask(mask_path: str, case_sensitive: bool = False, full_path: bool = False) → List[str]¶

Resolves a filemask with wildcard characters into a sorted list of matching file base names. Matching is case-insensitive by default.

Parameters:

mask_path (str) – Absolute path including a wildcard, e.g. "/data/exports/sales_*.csv".
case_sensitive (bool) – When True, match file names exactly as cased. Defaults to False.
full_path (bool) – When True, return full paths (folder of mask_path joined with each name). Defaults to False.

Returns:

Sorted list of matching file base names, or full paths when: full_path=True. Returns an empty list if no files match or the parent directory is missing.

Return type:

list[str]

validate_compatibility()¶: No additional compatibility checks are required for vanilla Spark.