Machine Logger

Structured, Delta-table-backed logging for operational observability.

class scamander.logging.machine_logger.MachineLogHandler(machine_logger: MachineLogger, level: int = 0)

Basisklassen: Handler

A logging.Handler that forwards HumanLogger records into a MachineLogger’s Delta table, so console log calls are also persisted.

Example:

log.addHandler(MachineLogHandler(machine_log))
log.info("This now also lands in the Delta logging table.")
emit(record: LogRecord) → None

Forward a stdlib LogRecord into the machine logger.

Called by the logging framework, not directly by SDK users. Appends the formatted exception traceback (if any) to the message before delegating to MachineLogger._log().

class scamander.logging.machine_logger.MachineLogger(notebook_name: str, notebook_path: str, session_id: str, logging_catalog: str, logging_schema: str, logging_table: str, log_level: str = 'INFO', timezone: str = 'Europe/Amsterdam', flush_interval_seconds: int = 300)

Basisklassen: object

Structured logger that persists log records to a Delta table for operational observability.

Each log call creates a single Spark Row and buffers it in memory rather than writing immediately — a synchronous Delta append per log line is prohibitively slow (a full Spark job + Delta commit each time). ERROR/EXCEPTION/CRITICAL calls trigger an immediate flush(), writing that row together with everything buffered before it, so a failure never loses the log trail leading up to it. Lower severities stay buffered until flush() is called explicitly, at process exit (via atexit), from the sys.excepthook safety net installed at construction time, or automatically every flush_interval_seconds via a background timer — a safety net so a long-running notebook never sits on an unflushed buffer for too long, without reintroducing per-log-line write overhead. The table is created automatically (with the correct schema) if it does not yet exist.

Parameters:

  • notebook_name (str) – Display name of the executing notebook.

  • notebook_path (str) – Browser-accessible URL of the executing notebook.

  • session_id (str) – UUID that groups all log records from one pipeline run.

  • logging_catalog (str) – Catalog name containing the logging schema.

  • logging_schema (str) – Schema name for the logging table.

  • logging_table (str) – Delta table name for log records.

  • log_level (str) – Minimum level a record must meet to be written to the table (e.g. "ERROR"). Records below this level are dropped before any Spark work happens. Defaults to "INFO".

  • timezone (str) – IANA timezone name used to stamp timestamp. Defaults to "Europe/Amsterdam".

  • flush_interval_seconds (int) – How often a background timer flushes the buffer automatically. Set to 0 to disable the timer and rely only on error-triggered/explicit/atexit flushes. Defaults to 300 (5 minutes).

Example:

machine_log = MachineLogger(
    notebook_name="ingest_customers",
    notebook_path="https://fabric.example.com/notebooks/ingest_customers",
    session_id=str(uuid.uuid4()),
    logging_catalog="main",
    logging_schema="mdif",
    logging_table="logging",
)
machine_log.info("Load started", object_name="silver.customers")
cleanup_old_records(retention_days: int) → None

Deletes log records older than retention_days from the Delta logging table.

Parameters:

retention_days (int) – Number of days of history to retain. Rows with a timestamp older than this are deleted.

Veroorzaakt:

ImportError – When the delta-spark package is not available.

critical(message: str, metadata_config_file: str = None, object_name: str = None, column_name: str = None, misc_parameters: str = None, status: str = None, total_records: int = None) → None

Logs a message at CRITICAL level.

debug(message: str, metadata_config_file: str = None, object_name: str = None, column_name: str = None, misc_parameters: str = None, status: str = None, total_records: int = None) → None

Logs a message at DEBUG level.

error(message: str, metadata_config_file: str = None, object_name: str = None, column_name: str = None, misc_parameters: str = None, status: str = None, total_records: int = None) → None

Logs a message at ERROR level.

exception(message: str, metadata_config_file: str = None, object_name: str = None, column_name: str = None, misc_parameters: str = None, status: str = None, total_records: int = None) → None

Logs a message at EXCEPTION level, automatically capturing and appending the current exception’s traceback when called from inside an except block.

flush() → None

Writes every buffered log row to the Delta table in a single append, then clears the buffer.

No-op when the buffer is empty. Safe to call multiple times (e.g. once from an ERROR-level log call and again at process exit), and safe to call concurrently with logging calls or the background flush timer.

info(message: str, metadata_config_file: str = None, object_name: str = None, column_name: str = None, misc_parameters: str = None, status: str = None, total_records: int = None) → None

Logs a message at INFO level.

warning(message: str, metadata_config_file: str = None, object_name: str = None, column_name: str = None, misc_parameters: str = None, status: str = None, total_records: int = None) → None

Logs a message at WARNING level.