JsonConnector

JSON / JSONL source connector.

class scamander.sources.json_connector.JsonConnector(config: Config, *args, **kwargs)

Basisklassen: Source

Source connector that reads JSON files into a Spark DataFrame.

Uses Spark’s built-in JSON reader, which supports:

  • Single JSON objects or arrays at the root level (multiline=True).

  • JSONL / NDJSON format — one JSON record per line (multiline=False, the default).

  • Directories containing multiple JSON files.

  • Glob patterns (e.g. "dbfs:/mnt/raw/events/*.json").

Spark reads all matching files into a single DataFrame, inferring a common schema across all records.

Can be instantiated directly with flat keyword arguments, or via the Source factory (Source(config, spark=spark)) by passing a Config whose source.options is a JsonSourceOptions.

Parameters:

  • file_path (str | Config) – Path, directory, or glob pattern to the JSON source(s), or a validated Config whose source.options is a JsonSourceOptions (the Source factory path — requires spark as a keyword argument).

  • multiline (bool) – True for multi-line JSON objects/arrays per file. False (default) for JSONL where each line is a record. Ignored when file_path is a Config.

  • infer_schema (bool) – Infer column types from the data. Defaults to True. Set to False to read all fields as STRING. Ignored when file_path is a Config.

  • encoding (str) – Character encoding of the files. Defaults to "UTF-8". Ignored when file_path is a Config.

  • spark (SparkSession) – Required keyword argument when file_path is a Config; unused otherwise.

Veroorzaakt:

ValueError – If file_path is a Config whose source.options is not a JsonSourceOptions instance.

connect() → DataFrame

Read the JSON file(s) into a Spark DataFrame.

Returns:

DataFrame with one row per JSON record (JSONL) or one row per top-level object/array element (multiline mode).