Databricks

Databricks platform implementation (Unity Catalog / DBFS).

class scamander.processor.databricks.Databricks(*args, **kwargs)

Basisklassen: Platform

Platform implementation for Databricks (Unity Catalog / DBFS environments).

Instantiated automatically by Platform.__new__ when the runtime is identified as Databricks (via DATABRICKS_RUNTIME_VERSION or DB_CLUSTER_ID environment variables). Uses DBUtils for all filesystem operations.

Example:

from scamander import Platform

# Running on a Databricks cluster: Platform() auto-detects and
# returns a Databricks instance transparently.
platform = Platform()
platform.dbutils.fs.ls("/mnt/raw")
check_file_exists(file_name: str, raise_exceptions=False) → bool

Checks whether a file or directory exists on the Databricks filesystem (DBFS).

Uses dbutils.fs.ls to probe the path; any exception is treated as “not found” rather than propagated (unless raise_exceptions=True).

Parameters:

  • file_name (str) – DBFS path to check, e.g. "dbfs:/mnt/raw/config.json".

  • raise_exceptions (bool) – When True, raises FileNotFoundError if the path does not exist instead of returning False. Defaults to False.

Returns:

True if the path exists, False otherwise.

Return type:

bool

Veroorzaakt:

FileNotFoundError – Only when raise_exceptions=True and the path is missing.

create_empty_table_from_metadata(metadata: dict)

Creates an empty Delta table from a metadata definition dictionary, including NOT NULL, PRIMARY KEY, and FOREIGN KEY constraints.

When metadata["track_history"] is True, five SCD Type-2 history-tracking columns are appended: HashSleutel, HashWaarde, GeldigVan, GeldigTot, IsActief.

Parameters:

metadata (dict) – Metadata dictionary with "table", "columns", and "track_history" keys.

get_dbutils()

Retrieves a DBUtils instance, supporting both interactive cluster and job cluster environments.

Attempts to import pyspark.dbutils.DBUtils first (job clusters / newer runtimes); falls back to reading dbutils from the IPython user namespace (interactive notebooks).

Returns:

An initialised Databricks Utilities object.

Return type:

DBUtils

get_notebook_name() → str

Returns the base name of the currently executing Databricks notebook.

Returns:

The notebook’s base name, e.g. "Ingest_S4SD".

Return type:

str

get_notebook_url() → str

Returns a browser-accessible URL to the currently executing Databricks notebook.

Returns:

A full HTTPS URL pointing to the notebook in the Databricks workspace UI.

Return type:

str

move_file(source: str, destination: str, overwrite=True, create_missing_dirs: bool = False) → bool

Moves a file from source to destination on DBFS using dbutils.fs.mv.

Parameters:

  • source (str) – DBFS path to the source file.

  • destination (str) – DBFS target path.

  • overwrite (bool) – If True (default), remove an existing destination file before moving.

  • create_missing_dirs (bool) – If True, create the destination directory if it does not already exist. Defaults to False.

Returns:

True on success, False on any failure.

Return type:

bool

resolve_filemask(mask_path: str, case_sensitive: bool = False, full_path: bool = False) → List[str]

Resolves a filemask with wildcard into a list of matching file names.

Splits the path on the last / to obtain the folder and glob pattern, lists the folder via dbutils.fs.ls, and returns names of files whose base name matches the pattern. Matching is case-insensitive by default.

Parameters:

  • mask_path (str) – Full path including a wildcard, e.g. "dbfs:/mnt/raw/sales_*.csv".

  • case_sensitive (bool) – When True, match file names exactly as cased. Defaults to False.

  • full_path (bool) – When True, return full paths (folder of mask_path joined with each name). Defaults to False.

Returns:

Sorted list of matching file base names, or full paths when

full_path=True. Returns an empty list if the folder does not exist or no files match.

Return type:

list[str]

validate_compatibility()

No additional compatibility checks are required for Databricks.