Getting started: Airflow¶

This page covers the Airflow product specifically: a customer-scoped Apache Airflow instance for scheduling production jobs. It’s an optional add-on to the platform — see Getting started: Sovereign Data Platform for the base Sovereign Data Platform (notebooks and querying), which Airflow jobs run against.

Management summary¶

It’s a dedicated, per-customer Apache Airflow instance, not a shared scheduler, for running the pipelines your organization already builds interactively in notebooks: unattended, on a schedule, in production.

Without it, “run this every night” means someone remembers to run it, or a cron job someone maintains by hand outside the platform, with no shared visibility into whether it actually ran. Airflow turns that into a small amount of declared, version-controlled code with built-in retry, alerting, and a shared view of what ran, when, and whether it succeeded.

Turning it on costs nothing extra to set up: it’s a flag (enableAirflow: true) on an already-provisioned customer environment, not a separate procurement. See the pricing page on the marketing site for what it adds to the bill.

What is Apache Airflow¶

Apache Airflow is an open-source workflow orchestrator. You declare what should run and in what order as Python code, and Airflow handles when: scheduling, retrying failures, and giving you one shared place to see what ran and what didn’t, instead of that logic being scattered across cron jobs and scripts on someone’s laptop.

Core concepts¶

DAG (Directed Acyclic Graph): A Python file declaring a set of tasks and the order they run in — see Developing DAGs for how to write one on this platform. “Acyclic” means dependencies only ever point forward: nothing can depend on something that (directly or indirectly) depends on it.
Task / Operator: A task is one node in the DAG; an operator is the reusable class that defines what that task actually does (run a notebook, call an API, wait for a file). Most DAG code is wiring existing operators together, not writing new logic.
Schedule: When a DAG’s runs are due — a cron expression, a preset like @daily, or (in modern Airflow) data-aware scheduling that triggers a DAG when another DAG produces a dataset it depends on.
DAG run / Task instance: One concrete execution of a DAG for a given schedule interval, made up of one task instance per task — each with its own state (queued, running, success, failed, retrying) and its own logs.
Scheduler and DAG processor: Background processes that parse your DAG files and decide what’s due to run next. You don’t interact with these directly, but “my DAG isn’t showing up” or “my DAG isn’t running on schedule” almost always traces back to how these two are parsing and scheduling it.
Connection: A named, centrally stored set of credentials/config a task uses to reach something external (a git repo, a database, an API) — see Airflow administrator guide: managing connections. DAG code references a connection by name; it never hardcodes the credential itself.
XCom: A small piece of data one task passes to another (a computed value, a file path) — not a bulk data-transfer mechanism.
Grid view / Graph view: The two main ways to look at a DAG’s run history and structure in the UI: Grid for status over time across runs, Graph for the dependency structure of a single run.

For the authoritative, versioned reference on all of this, see the official Apache Airflow documentation — in particular its own core concepts guide. This page and Developing DAGs only cover what’s specific to running Airflow on this platform.

How it fits on this platform¶

Identity¶

By default, Airflow authenticates against the same per-customer Keycloak realm as the rest of the platform (see Getting started: Sovereign Data Platform’s “Identity” section), so there’s no separate login. If your organization would rather manage Airflow access from its own Azure tenant, Airflow also supports authenticating directly against Microsoft Entra ID instead: see Onboarding: Microsoft Entra ID for the full onboarding walkthrough, and Managing access: Airflow roles for day-to-day role management once that’s set up.

Running your pipelines¶

The platform’s custom Airflow providers (see Custom Airflow providers) are what make “run this notebook as a production Spark job on a schedule” a small amount of DAG code rather than hand-writing Kubernetes pod specs. Airflow task pods and the Spark driver pods they create both run under a namespace-scoped spark-driver identity — a job for one customer can never reach into another customer’s namespace. Both the interactive (notebook) and scheduled paths share the same container image and the same Hive Metastore catalog, so there is no drift between how a job behaves in a notebook versus how it behaves in production.

Next steps¶

Get access — see Managing access: Airflow roles (or Onboarding: Microsoft Entra ID if this is the first time Entra ID sign-in is being set up).
Write your first DAG — see Developing DAGs, which covers how DAG code actually reaches this Airflow instance, the notebook-first development path, and platform-specific best practices.
Set up any external systems your DAGs need — see Airflow administrator guide: managing connections.