Core concepts

Platform auto-detection

Platform() reads environment variables to detect the runtime and returns the correct concrete subclass automatically. You never import Fabric or Databricks directly:

Environment variable	Detected platform
MMLSPARK_PLATFORM_INFO contains FABRIC or SYNAPSE	Fabric
DATABRICKS_RUNTIME_VERSION or DB_CLUSTER_ID present	Databricks
Neither	Spark (vanilla fallback, for local/CI use)

from scamander.processor.platform import Platform

platform = Platform()   # returns a Fabric, Databricks, or Spark instance

Config model

The Config Pydantic model is the top-level contract between your metadata files and the framework. It carries a source (what to read) and a sink (where to write):

from scamander.models import Config
from scamander.models.source_config import SourceConfig
from scamander.models.source_config.csv import CsvSourceOptions
from scamander.models.sink_config import SinkConfig
from scamander.models.sink_config.lakehouse import LakeHouseSinkOptions

config = Config(
    source=SourceConfig(type="csv", options=CsvSourceOptions()),
    sink=SinkConfig(type="LakeHouse", options=LakeHouseSinkOptions()),
)

In practice you parse Config from a JSON or YAML file rather than constructing it in code — see 2. Read a config file.

Metadata config files

Config files drive every pipeline run: sources, sinks, column schemas, and behaviour flags. JSON is the primary format; CSV and YAML are also supported.

A typical JSON metadata config for table creation:

{
  "table": "bronze.my_schema.customer",
  "track_history": false,
  "columns": [
    { "target": "customer_id",   "type": "INT",      "nullable": false, "primary_key": true },
    { "target": "customer_name", "type": "NVARCHAR",  "nullable": true  },
    { "target": "created_at",    "type": "DATETIME2", "nullable": false },
    { "target": "revenue",       "type": "DECIMAL",   "nullable": true, "precision": 18, "scale": 2 }
  ]
}