CsvSourceOptions¶

Connection options for the CSV source connector.

class scamander.models.source_config.csv.CsvSourceOptions(*, file_path: str, header: bool = True, separator: str = ';', encoding: str = 'UTF-8', infer_schema: bool = True, multiline: bool = False, date_format: str | None = None)¶

Bases: BaseModel

Connection options for the CSV source connector.

file_path¶

Absolute path to the CSV file or a glob pattern resolving to multiple CSV files (e.g. "dbfs:/mnt/raw/sales_*.csv").

Type:: str

header¶

Whether the first row of the file is a header row containing column names. Defaults to True.

Type:: bool

separator¶

The field delimiter character. Defaults to ";" (semicolon), which is standard in Dutch / European data exports.

Type:: str

encoding¶

Character encoding of the file. Defaults to "UTF-8".

Type:: str

infer_schema¶

When True, Spark samples the file to infer column data types. Set to False to read all columns as STRING (faster and avoids type-inference surprises). Defaults to True.

Type:: bool

multiline¶

Set to True when a single CSV record spans multiple lines (e.g. quoted fields containing newlines). Defaults to False.

Type:: bool

date_format¶

Optional java.text.SimpleDateFormat pattern used to parse date columns (e.g. "dd-MM-yyyy"). Only relevant when infer_schema=True.

Type:: str | None

model_config = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].