Hadoop Fs

Direct Hadoop FileSystem reads for small single-file access.

Bypasses launching a distributed Spark job (plan, scheduling, task dispatch) for reads that only ever touch one small file (a metadata config, an archive to copy locally) — the job-scheduling overhead otherwise dominates the actual I/O cost. Works uniformly across any Hadoop-compatible filesystem Spark already resolves paths through (OneLake, DBFS, ADLS, local), since it reuses the SparkSession’s own Hadoop configuration rather than a separate client.

scamander.utils.hadoop_fs.read_bytes(spark: SparkSession, path: str) → bytes

Read a file’s full contents directly via Spark’s underlying Hadoop FileSystem.

Parameters:

  • spark – Active SparkSession. Its JVM gateway and Hadoop configuration are reused to resolve path on whichever filesystem it belongs to.

  • path – Absolute path to the file, in the same form accepted by spark.read.

Returns:

The file’s full raw contents.

Return type:

bytes

Raises:

FileNotFoundError – If no file exists at path.

Example:

from scamander.utils.hadoop_fs import read_bytes

raw = read_bytes(platform.spark, "/lakehouse/default/Files/config.json")