Source

Source abstraction: factory dispatch to concrete source connectors by config type.

class scamander.sources.source.Source(config: Config, *args, **kwargs)

Bases: ABC

Abstract factory for data source connectors.

Calling Source(config) does not return a Source instance directly; instead __new__ inspects config.source.type and returns the appropriate concrete connector. This keeps call-site code agnostic of the underlying connector type.

Supported source types (config.source.type):
Parameters:

config (Config) – A validated Config model whose source.type determines which connector is instantiated.

Raises:

NotImplementedError – If config.source.type is not a recognised connector type.

Note

spark must always be passed as an explicit keyword argument alongside config — e.g. Source(config, spark=spark) — for every connector type; Source(config) without it raises TypeError: missing 1 required positional argument: 'spark'.

CsvConnector, JsonConnector, and ExcelConnector additionally support being constructed directly with flat keyword arguments, bypassing this factory entirely, for ad-hoc use without building a Config — in that case, pass file_path positionally: all three inherit this class’s __new__, whose first parameter is named config, so an all-keyword call raises TypeError: missing 1 required positional argument: 'config'. See the Example block below and the “Read data from a source” section of README.md for the full breakdown by connector.

Example:

from pyspark.sql import SparkSession
from scamander.sources import Source

spark = SparkSession.builder.getOrCreate()

# Every connector type: pass spark explicitly.
connector = Source(config, spark=spark)   # e.g. returns a PostgresConnector
df = connector.connect()

csv_connector = Source(csv_config, spark=spark)   # returns a CsvConnector
csv_df = csv_connector.connect()

# CSV / JSON / Excel additionally support direct construction; file_path
# must be positional in that case.
from scamander.sources.csv_connector import CsvConnector
direct_csv_connector = CsvConnector("dbfs:/mnt/raw/customers.csv", separator=",")
direct_csv_df = direct_csv_connector.connect()
abstractmethod connect() → DataFrame

Establishes a connection to the data source and returns its data as a Spark DataFrame.

Must be implemented by every concrete connector subclass.

Returns:

A Spark DataFrame containing the source data.

Return type:

DataFrame