Spark

Vanilla PySpark platform implementation (local development / CI).

class scamander.processor.spark.Spark(*args, **kwargs)

Basisklassen: Platform

Vanilla PySpark platform implementation for local development and CI environments.

Instantiated automatically by Platform.__new__ when neither Fabric nor Databricks environment variables are detected. Uses Python’s standard pathlib / shutil for filesystem operations, which works against any locally-mounted filesystem or object storage path that the OS can resolve.

Example:

from scamander import Platform

# Local dev / CI: no Fabric or Databricks env vars present, so
# Platform() auto-detects and returns a Spark instance transparently.
platform = Platform()
platform.spark.sql("SELECT 1").show()
check_file_exists(file_name: str, raise_exceptions=False) → bool

Checks whether a file or directory exists on the local / mounted filesystem.

Parameters:

  • file_name (str) – Absolute path to the file or directory.

  • raise_exceptions (bool) – When True, raises FileNotFoundError if the path does not exist. Defaults to False.

Returns:

True if the path exists, False otherwise.

Return type:

bool

Veroorzaakt:

FileNotFoundError – Only when raise_exceptions=True and the path is missing.

create_empty_table_from_metadata(metadata: dict)

Creates an empty Delta table from a metadata definition dictionary.

Vanilla Spark does not support PRIMARY KEY or FOREIGN KEY constraints in Delta DDL, so only column definitions and NOT NULL constraints are generated. When metadata["track_history"] is True, the five SCD Type-2 columns are appended automatically.

Parameters:

metadata (dict) – Metadata dictionary with "table", "columns", and "track_history" keys.

get_notebook_name() → str

Returns a placeholder notebook name for local Spark environments where no notebook context is available.

Returns:

The string "local" as a placeholder.

Return type:

str

get_notebook_url() → str

Returns a placeholder URL for local Spark environments where no notebook context is available.

Returns:

The string "local" as a placeholder.

Return type:

str

move_file(source: str, destination: str, overwrite=True, create_missing_dirs: bool = False) → bool

Moves a file from source to destination using shutil.move.

Parameters:

  • source (str) – Absolute path to the source file.

  • destination (str) – Absolute destination path.

  • overwrite (bool) – If True (default), remove an existing destination file before moving.

  • create_missing_dirs (bool) – If True, create the destination directory when it does not exist. Defaults to False.

Returns:

True on success, False on any failure.

Return type:

bool

resolve_filemask(mask_path: str, case_sensitive: bool = False, full_path: bool = False) → List[str]

Resolves a filemask with wildcard characters into a sorted list of matching file base names. Matching is case-insensitive by default.

Parameters:

  • mask_path (str) – Absolute path including a wildcard, e.g. "/data/exports/sales_*.csv".

  • case_sensitive (bool) – When True, match file names exactly as cased. Defaults to False.

  • full_path (bool) – When True, return full paths (folder of mask_path joined with each name). Defaults to False.

Returns:

Sorted list of matching file base names, or full paths when

full_path=True. Returns an empty list if no files match or the parent directory is missing.

Return type:

list[str]

validate_compatibility()

No additional compatibility checks are required for vanilla Spark.