apache-airflow-providers-notebook¶

Operators

SparkNotebookOperator runs a Jupyter notebook as a scheduled, cluster-mode Spark job: it checks out a notebook from a git repository, submits a SparkApplication custom resource (via SparkKubernetesOperator/ SparkKubernetesSensor under the hood), and the resulting driver pod runs the notebook through papermill_runner.py against the same platform image and Hive Metastore catalog interactive JupyterHub sessions use.

Usage¶

from airflow.providers.notebook.operators.spark_notebook import (
    SparkNotebookOperator,
)

run_notebook = SparkNotebookOperator(
    task_id="run_notebook",
    repo_url="https://dev.azure.com/<org>/<project>/_git/example-dags",
    repo_branch="main",
    notebook_path="open_data_notebook.ipynb",
    notebook_parameters='{"a": 1}',  # optional, passed to papermill
    # + optional driver/executor sizing knobs, extra_spark_conf
)

Everything beyond task_id/repo_url/repo_branch/notebook_path is deliberately not a DAG-settable parameter. The operator’s public surface was minimized aggressively so a DAG author can’t accidentally diverge from how the platform is actually deployed. Instead:

Setting	Source
Container image	`DATAPROCESSING_IMAGE` env var on the Airflow deployment
Kubernetes namespace	Auto-detected from the pod’s ServiceAccount projection — never DAG-settable
Spark version, Hive Metastore connection, checkout image, `imagePullSecrets`	Hardcoded module constants, one per Airflow deployment version
Delta Lake warehouse dir / S3A endpoint	`LAKE_WAREHOUSE_DIR` / `CUBEFS_S3_ENDPOINT` env vars on the Airflow deployment (required — the operator raises at construction if either is missing)
S3A access/secret key	A fixed Kubernetes Secret name (`cubefs-lakehouse-credentials`), wired into the driver/executor pod spec via `secretKeyRef`. It’s never read into Airflow’s own process environment, which keeps the blast radius of a credential leak scoped to this one Secret rather than every DAG’s Python code.
Git checkout credentials	The `git_default` Airflow connection (via `GitHook`), not a hand-created Secret

Both the task pod submitting the SparkApplication and the resulting driver pod run under the customer’s namespace-scoped spark-driver ServiceAccount, the same identity a JupyterHub notebook kernel uses in client mode. A customer’s Airflow instance has no wider reach into the cluster than that.

Requirements on the Airflow deployment¶

Anyone deploying Airflow with this provider needs to ensure, in the namespace Airflow’s workers run in:

Three env vars set on the Airflow deployment: DATAPROCESSING_IMAGE, LAKE_WAREHOUSE_DIR, CUBEFS_S3_ENDPOINT.
A Kubernetes Secret named cubefs-lakehouse-credentials with keys CUBEFS_ACCESS_KEY / CUBEFS_SECRET_KEY.
The spark-driver ServiceAccount with RBAC on sparkapplications and sparkapplications/status (a distinct sub-resource).
A git_default Airflow connection (type git) with read access to whatever repo_url a DAG passes.

See the platform’s own deployment repo for how these are provisioned.