LakeHouseConnector¶

Lakehouse / Unity Catalog Delta table source connector.

class scamander.sources.lakehouse_connector.LakeHouseConnector(config: Config, *args, **kwargs)¶

Basisklassen: Source

Source connector that reads a Delta table from a Fabric Lakehouse or Unity Catalog into a Spark DataFrame.

Supports optional column projection and row-level predicate pushdown.

Parameters:

config (Config) – Validated configuration object containing Lakehouse connection details under config.source.options.
spark (SparkSession) – An active Spark session.

Veroorzaakt:

ValueError – If config.source.options is not a LakeHouseSourceOptions instance.

connect() → DataFrame¶

Reads the target Delta table and returns it as a Spark DataFrame.

Applies column projection (options.columns) and filter predicate (options.filter) when set, pushing both down to the Delta reader for efficiency.