CsvConnector

CSV source connector.

class scamander.sources.csv_connector.CsvConnector(config: Config, *args, **kwargs)

Bases: Source

Source connector that reads one or more CSV files into a Spark DataFrame.

Can be instantiated directly with flat keyword arguments, or via the Source factory (Source(config, spark=spark)) by passing a Config whose source.options is a CsvSourceOptions.

Parameters:

  • file_path (str | Config) – Path to the CSV file or a glob pattern on the platform filesystem (e.g. a DBFS path, an ABFS URI, or a local path for testing), or a validated Config whose source.options is a CsvSourceOptions (the Source factory path — requires spark as a keyword argument).

  • header (bool) – Whether the first row of the CSV contains column names. Defaults to True. Ignored when file_path is a Config.

  • separator (str) – The field delimiter character. Defaults to ";" (semicolon). Ignored when file_path is a Config.

  • encoding (str) – Character encoding of the file. Defaults to "UTF-8". Ignored when file_path is a Config.

  • infer_schema (bool) – When True, Spark samples the file to infer column types. Defaults to True. Ignored when file_path is a Config.

  • multiline (bool) – Allow records that span multiple lines. Defaults to False. Ignored when file_path is a Config.

  • date_format (str | None) – java.text.SimpleDateFormat pattern for date columns. Ignored when file_path is a Config.

  • spark (SparkSession) – Required keyword argument when file_path is a Config; unused otherwise.

Raises:

ValueError – If file_path is a Config whose source.options is not a CsvSourceOptions instance.

connect() → DataFrame

Reads the CSV file(s) at self.file_path into a Spark DataFrame.

All reader options are forwarded to Spark’s CSV data source from the constructor arguments.

Returns:

A Spark DataFrame containing the CSV data.

Return type:

DataFrame