Archive Utils

Archive extraction utilities for local filesystem paths.

Supports .zip, .tar, .tar.gz / .tgz, and .tar.bz2 / .tbz2 using only Python stdlib modules — no extra dependencies required.

Path-traversal (zip-slip) attacks are blocked by resolving every member path and rejecting any entry whose destination would land outside the target directory.

scamander.utils.archive_utils.copy_spark_file_to_local(spark: SparkSession, source_path: str, local_path: str, log: HumanLogger | None = None) → str

Copy one Spark-readable binary file to the driver-local filesystem.

This is useful in notebook runtimes where archives live in a Lakehouse, DBFS, or object-store path, but Python stdlib archive tools need a local file path.

Parameters:

  • spark – Active SparkSession.

  • source_path – File path readable by Spark’s binaryFile source.

  • local_path – Driver-local destination path. Parent directories are created automatically.

  • log – Optional HumanLogger.

Returns:

The local_path that was written.

Veroorzaakt:

  • FileNotFoundError – When Spark finds no file at source_path.

  • ValueError – When source_path resolves to more than one file.

Example:

local_path = copy_spark_file_to_local(
    spark,
    "/lakehouse/default/Files/Inbox/export.tar.gz",
    "/tmp/export.tar.gz",
    log=platform.log,
)
scamander.utils.archive_utils.extract_archive(archive_path: str, destination: str, log: HumanLogger | None = None) → list[str]

Extract a zip or tar archive to a local directory.

The archive format is detected from the file extension. Every member path is validated before extraction to guard against path-traversal attacks — any member that would write outside destination raises ValueError.

Parameters:

  • archive_path – Absolute local path to the archive file.

  • destination – Directory to extract into. Created automatically when it does not yet exist.

  • log – Optional HumanLogger (platform.log). Extraction count and path are logged at INFO level when supplied.

Returns:

Sorted list of absolute paths of all extracted files (directories are excluded from the result).

Veroorzaakt:

  • FileNotFoundError – When archive_path does not exist.

  • ValueError – When the file extension is not a recognised archive format, or when the archive contains a path-traversal entry.

Example:

from scamander.utils import extract_archive

files = extract_archive(
    "/lakehouse/default/Files/Inbox/export.tar.gz",
    "/lakehouse/default/Files/Inbox/export/",
    log=platform.log,
)
# files → ["/lakehouse/default/Files/Inbox/export/data.json", ...]