Getting started
================

Prerequisites
-------------

.. list-table::
   :header-rows: 1

   * - Requirement
     - Version
     - Notes
   * - Python
     - ``>=3.10, <3.14``
     -
   * - Java
     - 8, 11, or 17
     - Required for local PySpark only; provided by Fabric / Databricks
   * - PySpark
     - ``>=3.4``
     - Provided by Fabric / Databricks; install locally for dev
   * - Delta Lake
     - ``>=3.0``
     - Provided by Fabric / Databricks; install locally via the ``[local]``
       extra

Installing PySpark locally
---------------------------

PySpark requires a Java runtime, then itself:

.. code-block:: bash

   # Linux (Ubuntu / Debian)
   sudo apt-get update && sudo apt-get install -y openjdk-17-jre-headless
   java -version
   pip install "pyspark>=3.4,<4.0" "delta-spark>=3.0"

On macOS, install OpenJDK via Homebrew (``brew install openjdk@17``) and
point ``JAVA_HOME`` at it. On Windows, install OpenJDK 17 from
`adoptium.net <https://adoptium.net>`_. If you're writing to the local
filesystem, also grab ``winutils.exe`` for your Hadoop version from
`github.com/cdarlint/winutils <https://github.com/cdarlint/winutils>`_.

Installation
------------

.. code-block:: bash

   # From the internal Azure Artifacts feed (recommended for pipeline use)
   pip install Scamander-MDIF

   # With platform-specific extras
   pip install "Scamander-MDIF[fabric]"
   pip install "Scamander-MDIF[databricks]"

   # AES-256-GCM encryption + Faker synthetic data for anonymization
   pip install "Scamander-MDIF[anonymization]"

   # Local development — PySpark, Delta Lake, anonymization, test tooling
   pip install "Scamander-MDIF[dev]"

In a Databricks or Fabric notebook:

.. code-block:: python

   %pip install Scamander-MDIF

Platform setup
--------------

**Microsoft Fabric**: before running any notebook,

1. Create a Lakehouse with **schemas enabled** (not the default). This is
   the logging destination.
2. Attach a default Lakehouse to the notebook.

The framework checks both on startup and raises a descriptive
``PlatformValidationException`` if either is missing.

**Databricks**: nothing extra needed beyond a running cluster with Unity
Catalog enabled.