apache-airflow-providers-git-azure ===================================== .. include:: _autodoc/git_azure_provider.inc.rst Adds ``AzureDevOpsGitDagBundle``: a `Git DAG bundle `_ for private Azure DevOps repositories with **no static credential**: no PAT, no SSH key, no client secret. This is what a DAG-processor/worker pod uses to actually *fetch* DAGs from git, as distinct from the other providers on this site, which add operators/hooks that a DAG's own tasks use. The bundle is **transparent**. It behaves exactly like the stock ``GitDagBundle`` for GitHub (or any other) URLs, PAT-based connections, and SSH-key connections, so it's safe to use as the bundle class for every git connection, including customer-managed ones. The Entra ID logic only activates for Azure DevOps HTTPS URLs (``dev.azure.com`` / ``*.visualstudio.com``): .. list-table:: :header-rows: 1 * - Connection contents (Azure DevOps HTTPS URL) - Auth mode * - Host only - Workload identity (``WorkloadIdentityCredential``, falling back to ``DefaultAzureCredential`` for local development) * - Login + password + ``tenant_id`` extra - Service principal client secret (``ClientSecretCredential``) * - Login + password, no ``tenant_id`` extra - PAT — stock ``GitHook`` behaviour * - SSH key (``private_key``/``key_file`` extra) or a ``git@…`` URL - Stock ``GitHook`` behaviour Why this exists ------------------ Azure DevOps Git over HTTPS accepts Microsoft Entra ID access tokens (audience ``499b84ac-1321-427f-aa17-267ca6975798``) as authentication, but they expire after roughly an hour: too short-lived to store in an Airflow connection. This bundle mints a fresh token at clone/fetch time instead, through the pod's `workload identity `_, so the connection only ever needs to carry the repository URL. Security properties: - The token is **never embedded in the remote URL**, so it's never persisted into the bare repository's ``.git/config``. - The token is passed to the git subprocess through ``GIT_CONFIG_COUNT``/``GIT_CONFIG_KEY_0=http.extraHeader``/ ``GIT_CONFIG_VALUE_0`` environment variables (git ≥ 2.31), not on the command line and not in any file. - The token is never logged, and the hook environment is cleared as soon as the git command finishes. Prerequisites ---------------- Workload identity only (skip if using a PAT or service principal client secret connection instead — see "Connection setup" below): 1. An AKS cluster with `workload identity `_ enabled, and a managed identity with federated credentials for the Airflow **dag-processor** and **worker** service accounts. 2. The managed identity added to the Azure DevOps organization (requires a **Basic** access level) and granted **Reader** + **Code: Read** on the target repository specifically (Project Settings → Repositories → ```` → Security) — not organization-wide — plus **View project-level information**, granted at the project level: Project Settings → Permissions → Users → search for the identity → expand its entry → enable **View project-level information**. 3. The Airflow pods labelled ``azure.workload.identity/use: "true"`` and the service account annotated with ``azure.workload.identity/client-id: ``. The service principal client secret flow has the same Azure DevOps-side requirement as step 2 above: the app registration itself (not a managed identity) must be onboarded to the org with a **Basic** access level and granted **Reader** + **Code: Read** on the repo, plus **View project-level information** on the project (Project Settings → Permissions → Users → search for the service principal → expand its entry). Connection setup ------------------- For a screenshot walkthrough of creating this connection in the Airflow UI, see :doc:`../connections/git_azure`. **Workload identity**: a standard ``git`` connection whose **host** is the HTTPS repository URL, with login/password left empty: .. code-block:: bash airflow connections add scamander-example \ --conn-type git \ --conn-host "https://dev.azure.com///_git/" **Service principal client secret**: the same connection, plus the service principal's client ID as login, its client secret as password, and the tenant in the extra. The password is used as an Entra client secret here, **not** as a git password. Without the ``tenant_id`` extra, a password is instead treated as a PAT: .. code-block:: bash airflow connections add scamander-example \ --conn-type git \ --conn-host "https://dev.azure.com///_git/" \ --conn-login "" \ --conn-password "" \ --conn-extra '{"tenant_id": ""}' DAG bundle configuration ---------------------------- Point the bundle config at the new class (``AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST`` or the corresponding Helm value): .. code-block:: json [ { "name": "scamander-example", "classpath": "airflow.providers.git_azure.bundles.git_azure.AzureDevOpsGitDagBundle", "kwargs": { "tracking_ref": "main", "git_conn_id": "scamander-example" } } ] ``AzureDevOpsGitDagBundle`` accepts the same kwargs as the stock ``GitDagBundle``: ``tracking_ref``, ``subdir``, ``git_conn_id``, ``repo_url``, ``submodules``, ``prune_dotgit_folder``, ``sparse_dirs``. Overriding the tracked branch per instance --------------------------------------------- A ``tracking_ref`` key in the connection's **extra** overrides the ``tracking_ref`` from the bundle config. That means a single deployed bundle config can track a different branch/tag per Airflow instance just by editing the connection: .. code-block:: bash airflow connections add scamander-example \ --conn-type git \ --conn-host "https://dev.azure.com///_git/" \ --conn-extra '{"tracking_ref": "release/2.0"}' The bundle config's ``tracking_ref`` kwarg is therefore optional. It's an error only if neither the config nor the connection provides a value.