Getting started¶

Prerequisites¶

Requirement	Version	Notes
Python	`>=3.10, <3.14`
Java	8, 11, or 17	Required for local PySpark only; provided by Fabric / Databricks
PySpark	`>=3.4`	Provided by Fabric / Databricks; install locally for dev
Delta Lake	`>=3.0`	Provided by Fabric / Databricks; install locally via the `[local]` extra

Installing PySpark locally¶

PySpark requires a Java runtime, then itself:

# Linux (Ubuntu / Debian)
sudo apt-get update && sudo apt-get install -y openjdk-17-jre-headless
java -version
pip install "pyspark>=3.4,<4.0" "delta-spark>=3.0"

On macOS, install OpenJDK via Homebrew (brew install openjdk@17) and point JAVA_HOME at it. On Windows, install OpenJDK 17 from adoptium.net. If you’re writing to the local filesystem, also grab winutils.exe for your Hadoop version from github.com/cdarlint/winutils.

Installation¶

# From the internal Azure Artifacts feed (recommended for pipeline use)
pip install Scamander-MDIF

# With platform-specific extras
pip install "Scamander-MDIF[fabric]"
pip install "Scamander-MDIF[databricks]"

# AES-256-GCM encryption + Faker synthetic data for anonymization
pip install "Scamander-MDIF[anonymization]"

# Local development — PySpark, Delta Lake, anonymization, test tooling
pip install "Scamander-MDIF[dev]"

In a Databricks or Fabric notebook:

%pip install Scamander-MDIF

Platform setup¶

Microsoft Fabric: before running any notebook,

Create a Lakehouse with schemas enabled (not the default). This is the logging destination.
Attach a default Lakehouse to the notebook.

The framework checks both on startup and raises a descriptive PlatformValidationException if either is missing.

Databricks: nothing extra needed beyond a running cluster with Unity Catalog enabled.