apache-airflow-providers-notebook
===================================

.. include:: _autodoc/notebook_provider.inc.rst

``SparkNotebookOperator`` runs a Jupyter notebook as a scheduled, cluster-mode
Spark job: it checks out a notebook from a git repository, submits a
``SparkApplication`` custom resource (via ``SparkKubernetesOperator``/
``SparkKubernetesSensor`` under the hood), and the resulting driver pod runs
the notebook through ``papermill_runner.py`` against the same platform image
and Hive Metastore catalog interactive JupyterHub sessions use.

Usage
-----

.. code-block:: python

   from airflow.providers.notebook.operators.spark_notebook import (
       SparkNotebookOperator,
   )

   run_notebook = SparkNotebookOperator(
       task_id="run_notebook",
       repo_url="https://dev.azure.com/<org>/<project>/_git/example-dags",
       repo_branch="main",
       notebook_path="open_data_notebook.ipynb",
       notebook_parameters='{"a": 1}',  # optional, passed to papermill
       # + optional driver/executor sizing knobs, extra_spark_conf
   )

Everything beyond ``task_id``/``repo_url``/``repo_branch``/``notebook_path``
is deliberately **not** a DAG-settable parameter. The operator's public
surface was minimized aggressively so a DAG author can't accidentally
diverge from how the platform is actually deployed. Instead:

.. list-table::
   :header-rows: 1

   * - Setting
     - Source
   * - Container image
     - ``DATAPROCESSING_IMAGE`` env var on the Airflow deployment
   * - Kubernetes namespace
     - Auto-detected from the pod's ServiceAccount projection — never
       DAG-settable
   * - Spark version, Hive Metastore connection, checkout image,
       ``imagePullSecrets``
     - Hardcoded module constants, one per Airflow deployment version
   * - Delta Lake warehouse dir / S3A endpoint
     - ``LAKE_WAREHOUSE_DIR`` / ``CUBEFS_S3_ENDPOINT`` env vars on the
       Airflow deployment (required — the operator raises at construction
       if either is missing)
   * - S3A access/secret key
     - A **fixed** Kubernetes Secret name (``cubefs-lakehouse-credentials``),
       wired into the driver/executor pod spec via ``secretKeyRef``. It's
       never read into Airflow's own process environment, which keeps the
       blast radius of a credential leak scoped to this one Secret rather
       than every DAG's Python code.
   * - Git checkout credentials
     - The ``git_default`` Airflow connection (via ``GitHook``), not a
       hand-created Secret

Both the task pod submitting the ``SparkApplication`` and the resulting
driver pod run under the customer's namespace-scoped ``spark-driver``
ServiceAccount, the same identity a JupyterHub notebook kernel uses in
client mode. A customer's Airflow instance has no wider reach into the
cluster than that.

Requirements on the Airflow deployment
---------------------------------------

Anyone deploying Airflow with this provider needs to ensure, in the
namespace Airflow's workers run in:

- Three env vars set on the Airflow deployment: ``DATAPROCESSING_IMAGE``,
  ``LAKE_WAREHOUSE_DIR``, ``CUBEFS_S3_ENDPOINT``.
- A Kubernetes Secret named ``cubefs-lakehouse-credentials`` with keys
  ``CUBEFS_ACCESS_KEY`` / ``CUBEFS_SECRET_KEY``.
- The ``spark-driver`` ServiceAccount with RBAC on ``sparkapplications``
  *and* ``sparkapplications/status`` (a distinct sub-resource).
- A ``git_default`` Airflow connection (type ``git``) with read access to
  whatever ``repo_url`` a DAG passes.

See the platform's own deployment repo for how these are provisioned.