Airflow administrator guide: managing connections ===================================================== Airflow connections that authenticate with a ``client_id``/``client_secret`` (Databricks, Fabric, Power BI, Synapse, Data Factory, a private git repo, or any generic OAuth-protected API) need that secret rotated before it expires. The **Airflow Connections Pipeline** is a self-contained Azure DevOps pipeline package that does this automatically, on a schedule, for every connection you list, without ever handing out Airflow's own application secret. Overview --------- One pipeline, one schedule, a list of connections. On every scheduled run, the pipeline loops over the list and, per entry: 1. Rotates the **target** app registration's client secret: the app that owns the thing Airflow is connecting to (a Databricks service principal, a Fabric service principal, a generic API's app registration, and so on). Name the target by ``targetAppName`` to have the pipeline create the app registration and service principal when one is missing (the recommended way to give every connection its own dedicated SP), or by ``targetAppId`` to rotate an existing one. A freshly created SP still needs its resource access (workspace roles, ARM RBAC) granted once, manually. 2. Ensures the service connection's SP holds the Airflow app's "Admin" app role, self-granting it via Microsoft Graph when missing. 3. Mints an access token for the Airflow application itself, without ever using the Airflow app's own client secret (see "One service connection" below). 4. Pushes the new ``client_id``/``client_secret`` into the named Airflow connection via the Airflow REST API. Because every entry rotates on the same schedule, pick a cron cadence shorter than the smallest ``validityDays`` across all your connections. Getting the pipeline ---------------------- This is a self-contained Azure DevOps pipeline package, with no dependency on the rest of the platform repo. Get a working copy one of two ways: - :download:`Download the pipeline package (zip) ` — the template, the scripts, and an example ``connections.yaml`` to copy and edit by hand. - :doc:`Generate a starting connections.yaml ` — fill in a form and get a ready-to-drop-in file for the connections you actually need, instead of hand-editing the example. Either way, unzip the package into its own Azure DevOps repo (or a folder of an existing one) and point a new pipeline at it — see "Setting up your pipeline" below. One service connection ------------------------ Each entry names a single Azure DevOps service connection (``serviceConnectionName``, typically the same one for every entry) whose service principal does everything: .. list-table:: :header-rows: 1 * - Job - What the SP needs * - Finds/creates target apps and rotates their secrets - Owner of those app registrations. To let the pipeline *create* per-connection apps (``targetAppName``), the Graph application permission ``Application.ReadWrite.OwnedBy`` — apps it creates are automatically owned by it. * - Authenticates to the Airflow REST API - The "Admin" app role on the Airflow application — self-granted by the pipeline when the SP holds ``AppRoleAssignment.ReadWrite.All``, otherwise granted once manually. You will never need, and should never be given, the Airflow application's own client secret. Instead, the service connection's SP gets the Airflow app's "Admin" app role. When the pipeline requests a token for the Airflow app as the resource/audience, Microsoft Entra ID includes that role in a ``roles`` claim regardless of which client asked for the token. Airflow's auth layer only checks the claim, not who's holding the app's secret. If the SP can't self-grant the role (missing ``AppRoleAssignment.ReadWrite.All``), the pipeline fails with a clear error and two resolution paths: someone with rights over the Airflow app registration runs ``Grant-AirflowAdminRole.ps1`` once, or the service connection's SP is granted that Graph permission (admin consent required) so the pipeline becomes self-sufficient on every future run. Either way, a freshly granted role can take a short while to propagate — re-run if the very next step still gets a 403. Setting up your pipeline in Azure DevOps -------------------------------------------- Everything below happens in **your own** organization's Azure DevOps, not the vendor's. No prior Azure DevOps experience is assumed: every step names the exact menu to click. 1. Get the files into a repo ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. In your Azure DevOps project (left sidebar), click **Repos**. 2. If the project doesn't have a repo yet, Azure DevOps offers to create one on that same screen — accept the default, or give it a name like ``airflow-connections``. If you'd rather keep this pipeline in an existing repo instead of a new one, that's fine too; any subfolder works. 3. Add the pipeline's files to that repo, keeping the folder structure (``templates/``, ``scripts/``, and your ``connections.yaml``) intact: - **No git experience needed**: on the Repos → Files screen, click **⋯** (top right) → **Upload file(s)**, and repeat per folder (``templates/rotate-connection-template.yaml``, everything under ``scripts/``, and your ``connections.yaml``). - **If you already use git**: clone the repo, copy the unzipped package's contents in, ``git add`` / ``git commit`` / ``git push``. 2. Fill in your connections.yaml ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Start from ``examples/connections.yaml`` in the downloaded package, or use :doc:`the generator ` to produce one directly. 2. For each connection you need, add an entry to the ``connections`` list — see the connection type reference below. Use ``targetAppName`` to have the pipeline create a dedicated app registration + SP per connection, or ``targetAppId`` to pin an existing one. 3. ``connectionId`` is the Airflow ``conn_id`` and must be unique within the file — letters, digits, underscores, and hyphens only. 4. Pick a ``schedules:`` cron cadence shorter than the smallest ``validityDays`` among your connections, so every secret gets rotated before it expires. 3. Create the service connection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is the one identity the pipeline runs as (see "One service connection" above) — its name must match the ``serviceConnectionName`` variable in your YAML file exactly. .. note:: The Airflow application itself belongs to the platform and lives in the platform's own tenant, not yours. Self-granting it the "Admin" role (below) only works once your organization has completed :doc:`Entra ID onboarding ` — that's what consents the platform's Airflow application into *your* tenant, which is what your service connection's SP needs to be able to assign it a role at all. If your Airflow instance is instead secured through the platform's shared Keycloak realm, this rotation mechanism's Airflow step doesn't apply — talk to the platform team about rotating that connection's secret. 1. Bottom-left corner of Azure DevOps, click **Project Settings**. 2. Under **Pipelines**, click **Service connections**. 3. Click **New service connection**. 4. Choose **Azure Resource Manager**, then click **Next**. 5. Choose **Service principal (automatic)** — Azure DevOps creates the app registration for you, so there's no separate Entra ID step for this part. 6. Pick your **Subscription** (any subscription in your own tenant — this only matters for creating the service principal itself; which one you pick doesn't affect its permissions, since everything this pipeline needs is granted at the Microsoft Entra ID level, not the subscription level). Leave **Resource group** empty (scope: **Subscription**). 7. Under **Service connection name**, enter exactly the same name as ``serviceConnectionName`` in your YAML file (e.g. ``customer-connections``). 8. **Uncheck** **Grant access permission to all pipelines**. This service connection ends up holding sensitive rights (ownership of your target apps, the Graph permissions below) — don't leave every pipeline in the project free to use it. Click **Save**. The first time you run the pipeline you create in the next section, Azure DevOps will show a banner asking to authorize that specific pipeline to use this connection — approve it there. The service connection now exists, but it only has the base subscription role Azure DevOps gave it. It still needs two Microsoft Graph **application permissions** before the pipeline can do its job. Grant these once, as a Microsoft Entra ID administrator: 9. Open `portal.azure.com `_, then **Microsoft Entra ID** → **App registrations**. 10. Find the app registration Azure DevOps just created — back on the service connection you made (**Project Settings** → **Service connections** → your connection → **Manage Service Principal**) opens it directly in the Entra portal. 11. Click **API permissions** → **Add a permission** → **Microsoft Graph** → **Application permissions**. 12. Search for and check both: - ``Application.ReadWrite.OwnedBy`` — needed if any connection uses ``targetAppName`` (lets the pipeline create that connection's app registration). - ``AppRoleAssignment.ReadWrite.All`` — needed so the pipeline can self-grant itself the Airflow "Admin" role (see "Granting the Admin role" above). Click **Add permissions**. 13. Click **Grant admin consent for **, then **Yes** to confirm. The permissions row should now show a green checkmark under **Status**. If you don't use ``targetAppName`` anywhere in your file, you can skip ``Application.ReadWrite.OwnedBy``. If you'd rather not grant ``AppRoleAssignment.ReadWrite.All`` at all, skip it too — the pipeline will instead print manual-grant instructions the first time it runs (see "Granting the Admin role" above, option (a)). 4. Create the pipeline ~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Left sidebar, click **Pipelines** → **New pipeline**. 2. **Where is your code?** → **Azure Repos Git** → select the repo you pushed the files into. 3. **Configure your pipeline** → **Existing Azure Pipelines YAML file**. 4. **Branch**: ``main`` (or whichever branch you pushed to). **Path**: pick your ``connections.yaml`` file (e.g. ``/customer-connections.yaml``). 5. Click **Continue**, review the YAML preview, then click the dropdown next to **Run** and choose **Save** (not **Save and run** — see the next step first). 6. Optional: rename the pipeline to something recognizable — open it, click **⋯** (top right) → **Rename/move**, e.g. "Rotate Airflow connections". 5. Run it once, then trust the schedule ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Open the pipeline and click **Run pipeline** → **Run**. 2. Watch each step. If the admin-role step fails, it prints the exact two fix options — see "Granting the Admin role" above and Troubleshooting below. 3. Once every step is green, your connections are live in Airflow — check the Airflow UI's **Admin → Connections** page for the ``connectionId``(s) you configured. 4. Nothing else to do: the ``schedules:`` block in your YAML already arms recurring runs. Confirm the next run time under the pipeline's **Scheduled runs** (clock icon, top right of the pipeline's page). Connection type reference ---------------------------- The pipeline template dispatches on ``connType``. Types with a dedicated script take typed fields directly (no hand-written ``extraJson``); any other ``connType`` falls through to the generic ``Set-AirflowConnection.ps1`` step with ``hostName``/``extraJson``. Login/password are always the rotated ``client_id``/``client_secret``, and the tenant ID is derived at runtime from the service connection's Azure context, never spelled out in YAML. .. list-table:: :header-rows: 1 * - Type - ``connType`` - Typed fields - Notes * - HTTP (generic OAuth-protected API) - ``http`` - ``hostName``, ``extraJson`` (``tenant_id``, ``scope``, ``token_url``) - Airflow's built-in ``HttpHook`` doesn't do client-credentials OAuth itself — pair this with a small custom hook in your DAGs that reads these fields and performs the token exchange. * - Databricks - ``databricks`` - ``hostName`` (workspace URL) - Matches ``apache-airflow-providers-databricks``'s documented Azure AD SPN auth method (extra ``azure_tenant_id``). * - Fabric (``FabricOperator``) - ``fabric`` - ``delegatedScopes`` (``User.Read Workspace.ReadWrite.All Item.ReadWrite.All Item.Execute.All``) - Extra ``tenantId``; workspace/item IDs are operator arguments, not connection fields. Scopes are declared on the target app's API permissions by the rotate step — admin consent is a one-time manual step. * - Fabric Capacity (``FabricCapacityOperator``) - ``fabric_capacity`` - — - Extra ``tenantId``; subscription/resource group/capacity name are operator arguments. Targets the ARM management plane — assign the SP the custom role from "Fabric Capacity access" below. * - Power BI - ``power_bi`` - — - Extra ``tenant_id``; dataset/group IDs are operator arguments. See :doc:`providers/powerbi_provider`. * - Synapse (``AzureSynapseHook``) - ``azure_synapse`` - ``hostName`` (workspace URL), ``subscriptionId`` - Extras ``tenantId``/``subscriptionId``. ``AzureSynapsePipelineHook`` reads the same fields, so one connection serves both. * - Data Factory (``AzureDataFactoryHook``) - ``azure_data_factory`` - ``subscriptionId``, ``resourceGroup``, ``factoryName`` - Extras ``tenantId``/``subscriptionId`` plus ``resource_group_name``/``factory_name``; resource group/factory are connection-level defaults operators can override. * - Git (private DAGs repo) - ``git`` - ``repoUrl`` (HTTPS clone URL), ``trackingRef`` (branch) - Same dedicated-app-per-connection model — use ``targetAppName`` to give this connection its own SP, then see `Git: authorizing the DAGs repo (one-time)`_ below before expecting it to actually work. See also :doc:`providers/git_azure_provider` for the workload-identity alternative that needs no rotated secret at all. Git: authorizing the DAGs repo (one-time) ----------------------------------------------- Rotating the ``git`` connection's secret only keeps its service principal's credentials valid. It does **not**, by itself, give that service principal permission to read the DAGs repo. If the DAGs repo lives in Azure DevOps and you're using ``targetAppName`` to give this connection its own dedicated service principal, do the following once per service principal, as an Azure DevOps organization administrator, **before** the pipeline's first run against that connection: 1. **Give the service principal an access level (a "license").** Azure DevOps requires one for every identity that touches it, service principals included — without this step, the repo permission you set up in step 2 exists but Git operations still fail. a. **Organization Settings** (gear icon, bottom-left) → **Users** → **Add users**. b. In the users field, paste the service principal's **Application (client) ID** (found on the app registration's **Overview** page in Microsoft Entra ID). c. **Access level**: choose **Basic**. **Stakeholder** does *not* include Git read access to a private repo, so it won't work here. d. **Add to project(s)**: pick the project the DAGs repo lives in. Click **Add**. 2. **Authorize it on the specific repo — not organization-wide.** a. **Project Settings** → **Repositories** → select the DAGs repo → **Security** tab. b. Click **Add**, and paste the same service principal identity. c. Set its permissions to **Reader** (covers **Read** / **Code: Read** — everything this connection type needs, since it only clones and pulls, never pushes). Repeat both steps for every ``git`` connection entry that has its own dedicated service principal. If several ``git`` connections share one target app (via ``targetAppId`` instead), you only need to do this once for that shared identity. Fabric Capacity access (one-time) ------------------------------------- The ``fabric_capacity`` connection drives suspend/resume/scale through the ARM management plane, so its SP needs a role on the capacity resource, not Contributor. Create a minimal custom role once (``Fabric Capacity (Smooth) Operator``) granting ``Microsoft.Fabric/capacities/{read,write}``, ``capacities/suspend/action``, ``capacities/resume/action``, and ``capacities/skus/read`` / ``skus/read``, then assign it to the pipeline-created SP for that connection (run the pipeline once first so the SP exists). Leave out ``write`` if you only ever suspend/resume, not scale. Troubleshooting ----------------- .. list-table:: :header-rows: 1 * - Symptom - Likely cause * - Can't reach the Airflow API (after retries) - ``airflowBaseUrl`` is wrong or the instance is down — not an auth problem, since this step retries with backoff first. * - Admin-role step fails with manual-resolution steps - The service connection's SP can't self-grant the Airflow "Admin" role — grant it manually or give the SP ``AppRoleAssignment.ReadWrite.All``, then re-run. * - 401/403 from the Airflow API even though the admin-role step passed - A freshly granted role can take a short while to propagate — re-run. If it persists, confirm the resolved Airflow app is the right one (pin it explicitly with ``airflowAppId`` if ``airflowAppName`` resolves ambiguously). * - "does not exist" rotating the target secret - With ``targetAppId`` the target app must already exist — switch to ``targetAppName`` if you want the pipeline to create a dedicated app per connection. * - Can't create the target app - The service connection's SP lacks app-creation rights — either have an admin create the app manually and pin it via ``targetAppId``, or grant the SP ``Application.ReadWrite.OwnedBy`` and re-run. * - Pipeline succeeded but the target system returns 401/403 to DAGs - A freshly created per-connection SP has credentials but no access yet — grant it the resource-side access once (workspace role, ARM RBAC, etc.).