Core concepts
==============

Platform auto-detection
------------------------

``Platform()`` reads environment variables to detect the runtime and returns
the correct concrete subclass automatically. You never import ``Fabric``
or ``Databricks`` directly:

.. list-table::
   :header-rows: 1

   * - Environment variable
     - Detected platform
   * - ``MMLSPARK_PLATFORM_INFO`` contains ``FABRIC`` or ``SYNAPSE``
     - ``Fabric``
   * - ``DATABRICKS_RUNTIME_VERSION`` or ``DB_CLUSTER_ID`` present
     - ``Databricks``
   * - Neither
     - ``Spark`` (vanilla fallback, for local/CI use)

.. code-block:: python

   from scamander.processor.platform import Platform

   platform = Platform()   # returns a Fabric, Databricks, or Spark instance

Config model
------------

The ``Config`` Pydantic model is the top-level contract between your
metadata files and the framework. It carries a ``source`` (what to read)
and a ``sink`` (where to write):

.. code-block:: python

   from scamander.models import Config
   from scamander.models.source_config import SourceConfig
   from scamander.models.source_config.csv import CsvSourceOptions
   from scamander.models.sink_config import SinkConfig
   from scamander.models.sink_config.lakehouse import LakeHouseSinkOptions

   config = Config(
       source=SourceConfig(type="csv", options=CsvSourceOptions()),
       sink=SinkConfig(type="LakeHouse", options=LakeHouseSinkOptions()),
   )

In practice you parse ``Config`` from a JSON or YAML file rather than
constructing it in code — see :ref:`mdif-read-config`.

Metadata config files
-----------------------

Config files drive every pipeline run: sources, sinks, column schemas, and
behaviour flags. JSON is the primary format; CSV and YAML are also
supported.

A typical JSON metadata config for table creation:

.. code-block:: json

   {
     "table": "bronze.my_schema.customer",
     "track_history": false,
     "columns": [
       { "target": "customer_id",   "type": "INT",      "nullable": false, "primary_key": true },
       { "target": "customer_name", "type": "NVARCHAR",  "nullable": true  },
       { "target": "created_at",    "type": "DATETIME2", "nullable": false },
       { "target": "revenue",       "type": "DECIMAL",   "nullable": true, "precision": 18, "scale": 2 }
     ]
   }