Developing DAGs

This page is for anyone writing pipeline code for this platform’s Airflow instance: what a DAG is, how your code actually gets onto the instance, how to develop and test one, and the practices that keep DAGs from breaking each other or from breaking on their own.

What is a DAG

A DAG (Directed Acyclic Graph) is Airflow’s unit of scheduling: a Python file that declares a set of tasks and the order they must run in — “run B after A succeeds,” never a cycle back to something already run. Airflow reads that file on a schedule, and for each due run, executes the tasks in dependency order, retrying or alerting on failure per task. The DAG file itself does not do the work — it wires together operators, each of which knows how to do one kind of task (run a notebook, call an API, wait for a file). See Custom Airflow providers for the operators this platform adds on top of Airflow’s built-ins.

How your DAG code reaches this Airflow instance

There is no manual deploy step and no file copying. This platform’s Airflow instances are configured with a Git DAG bundle (apache-airflow-providers-git-azure) that fetches your DAG repository directly — for Azure DevOps repos, with no static credential (PAT/SSH key) required at all, via workload identity. The bundle tracks one branch (or tag) per Airflow instance, so:

  • Pushing to that tracked branch (typically main) is what “deploys” a DAG. There’s nothing else to run.

  • A non-production Airflow instance can track a different branch by overriding tracking_ref on its own connection (see apache-airflow-providers-git-azure’s “Overriding the tracked branch” section). Use this to test a DAG before it reaches an instance people depend on.

  • New tasks, changed schedules, anything in the DAG file: all picked up automatically the next time the bundle syncs. No Airflow restart needed.

Developing a DAG

Start from a notebook, if the job is data work. The platform’s apache-airflow-providers-notebook (SparkNotebookOperator) runs a Jupyter notebook as a scheduled, cluster-mode Spark job — the same notebook you can already run interactively in JupyterHub, against the same image and the same Delta Lake catalog. This is the intended path for “turn analysis into a production job”: write and validate the notebook interactively first, then wrap it in a one-task DAG that runs it on a schedule. You should rarely need to hand-write Spark submission logic.

from datetime import datetime
from airflow import DAG
from airflow.providers.notebook.operators.spark_notebook import (
    SparkNotebookOperator,
)

with DAG(
    dag_id="daily_open_data_refresh",
    schedule="@daily",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    tags=["open-data"],
) as dag:
    SparkNotebookOperator(
        task_id="run_notebook",
        repo_url="https://dev.azure.com/<org>/<project>/_git/example-dags",
        repo_branch="main",
        notebook_path="pipelines/open_data_notebook.ipynb",
    )

For non-notebook work, use the other platform providers (Custom Airflow providers) or Airflow’s own upstream providers where they apply — for example git_default-style connections for repo access, or the mail/Fabric/Power BI/SSIS providers this platform ships for those specific integrations. Reach for a plain PythonOperator/@task only when nothing else fits; anything running Spark work should go through SparkNotebookOperator rather than a hand-rolled submission, so it picks up the platform’s image, catalog, and credential wiring automatically instead of re-deriving it per DAG.

Testing before it reaches an instance people rely on. Point a non-production Airflow instance’s connection at your feature branch (see above), let it sync, and confirm the DAG parses and runs there — the DAG import errors panel (visible to Airflow Admin, see Managing access: Airflow roles) is the fastest way to catch a syntax or import error without waiting for a scheduled run. Only merge to the tracked production branch once it runs clean.

Best practices

Tasks get retried, and runs get cleared and re-triggered by hand, so design them to be idempotent: running the same task twice with the same inputs should produce the same result, not a duplicate row or a corrupted state. Overwriting or upserting a partition works; appending to it doesn’t.

Don’t put an access key, password, or token in a DAG file, not even as an “example.” Read it from an Airflow connection instead. The platform’s own operators are built the same way: SparkNotebookOperator doesn’t accept a credential-shaped parameter at all (see apache-airflow-providers-notebook for what’s fixed at the deployment level instead of left DAG-settable), so there’s no way to accidentally diverge from how the platform is actually secured.

Set catchup=False unless you actually need a backfill. Without it, a DAG with a start_date in the past will trigger a run for every missed interval since then, all at once, the moment it’s deployed — rarely what you want for an operational pipeline.

Give every DAG retries and a sane timeout, either at the task level (retries, retry_delay) or via default_args for the whole DAG. A slow upstream API or a momentary resource crunch shouldn’t need a human to notice and re-trigger the run by hand.

Name DAGs and tag them so ownership is obvious at a glance. dag_id and tags are the only things another team sees when your DAG turns up failing in the shared Grid view, so use something team- or domain-prefixed instead of a generic pipeline or test.

Keep one DAG’s blast radius to one DAG. Two DAGs writing to the same table on independent schedules is a race condition waiting to happen — if they genuinely share a dependency, model it explicitly (a sensor, or a dataset-aware schedule) rather than leaving it to the accident of timing.

And check the Grid/Graph view and task logs before assuming a fix worked. A DAG parsing and a DAG producing correct output are two different guarantees; Airflow only checks the first one for you automatically.