ExcelConnector

Excel source connector.

class scamander.sources.excel_connector.ExcelConnector(config: Config, *args, **kwargs)

Basisklassen: Source

Source connector that reads a single Excel worksheet into a Spark DataFrame.

Reads the worksheet with pandas (via the openpyxl engine) and converts the result to a Spark DataFrame. Can be instantiated directly with flat keyword arguments, or via the Source factory (Source(config, spark=spark)) by passing a Config whose source.options is an ExcelSourceOptions.

Parameters:

  • file_path (str | Config) – Path to the Excel file on the platform filesystem (e.g. a DBFS path, an ABFS URI, or a local path for testing), or a validated Config whose source.options is an ExcelSourceOptions (the Source factory path — requires spark as a keyword argument).

  • sheet_name (str | int) – Name or zero-based index of the worksheet to read. Defaults to 0 (the first sheet). Ignored when file_path is a Config.

  • header (bool) – Whether the first row (after skip_rows) contains column names. Defaults to True. Ignored when file_path is a Config.

  • skip_rows (int) – Number of rows to skip before the header/data begins. Defaults to 0. Ignored when file_path is a Config.

  • infer_schema (bool) – When True, pandas infers column data types from the sheet contents. Defaults to True. Set to False to read all columns as STRING. Ignored when file_path is a Config.

  • spark (SparkSession) – Required keyword argument when file_path is a Config; unused otherwise.

Veroorzaakt:

ValueError – If file_path is a Config whose source.options is not an ExcelSourceOptions instance.

connect() → DataFrame

Reads the configured worksheet at self.file_path into a Spark DataFrame.

Reads the sheet with pandas, then converts the resulting pandas DataFrame into a Spark DataFrame via spark.createDataFrame.

Returns:

A Spark DataFrame containing the worksheet data.

Return type:

DataFrame