Airflow administrator guide: managing connections

Airflow connections that authenticate with a client_id/client_secret (Databricks, Fabric, Power BI, Synapse, Data Factory, a private git repo, or any generic OAuth-protected API) need that secret rotated before it expires. The Airflow Connections Pipeline is a self-contained Azure DevOps pipeline package that does this automatically, on a schedule, for every connection you list, without ever handing out Airflow’s own application secret.

Overview

One pipeline, one schedule, a list of connections. On every scheduled run, the pipeline loops over the list and, per entry:

1. Rotates the target app registration’s client secret: the app that owns the thing Airflow is connecting to (a Databricks service principal, a Fabric service principal, a generic API’s app registration, and so on). Name the target by targetAppName to have the pipeline create the app registration and service principal when one is missing (the recommended way to give every connection its own dedicated SP), or by targetAppId to rotate an existing one. A freshly created SP still needs its resource access (workspace roles, ARM RBAC) granted once, manually.

2. Ensures the service connection’s SP holds the Airflow app’s “Admin” app role, self-granting it via Microsoft Graph when missing.

3. Mints an access token for the Airflow application itself, without ever using the Airflow app’s own client secret (see “One service connection” below).

4. Pushes the new client_id/client_secret into the named Airflow connection via the Airflow REST API.

Because every entry rotates on the same schedule, pick a cron cadence shorter than the smallest validityDays across all your connections.

Getting the pipeline

This is a self-contained Azure DevOps pipeline package, with no dependency on the rest of the platform repo. Get a working copy one of two ways:

• Download the pipeline package (zip) — the template, the scripts, and an example connections.yaml to copy and edit by hand.

• Generate a starting connections.yaml — fill in a form and get a ready-to-drop-in file for the connections you actually need, instead of hand-editing the example.

Either way, unzip the package into its own Azure DevOps repo (or a folder of an existing one) and point a new pipeline at it — see “Setting up your pipeline” below.

One service connection

Each entry names a single Azure DevOps service connection (serviceConnectionName, typically the same one for every entry) whose service principal does everything:

Job	What the SP needs
Finds/creates target apps and rotates their secrets	Owner of those app registrations. To let the pipeline create per-connection apps (targetAppName), the Graph application permission Application.ReadWrite.OwnedBy — apps it creates are automatically owned by it.
Authenticates to the Airflow REST API	The “Admin” app role on the Airflow application — self-granted by the pipeline when the SP holds AppRoleAssignment.ReadWrite.All, otherwise granted once manually.

You will never need, and should never be given, the Airflow application’s own client secret. Instead, the service connection’s SP gets the Airflow app’s “Admin” app role. When the pipeline requests a token for the Airflow app as the resource/audience, Microsoft Entra ID includes that role in a roles claim regardless of which client asked for the token. Airflow’s auth layer only checks the claim, not who’s holding the app’s secret.

If the SP can’t self-grant the role (missing AppRoleAssignment.ReadWrite.All), the pipeline fails with a clear error and two resolution paths: someone with rights over the Airflow app registration runs Grant-AirflowAdminRole.ps1 once, or the service connection’s SP is granted that Graph permission (admin consent required) so the pipeline becomes self-sufficient on every future run. Either way, a freshly granted role can take a short while to propagate — re-run if the very next step still gets a 403.

Setting up your pipeline in Azure DevOps

Everything below happens in your own organization’s Azure DevOps, not the vendor’s. No prior Azure DevOps experience is assumed: every step names the exact menu to click.

1. Get the files into a repo

1. In your Azure DevOps project (left sidebar), click Repos.

2. If the project doesn’t have a repo yet, Azure DevOps offers to create one on that same screen — accept the default, or give it a name like airflow-connections. If you’d rather keep this pipeline in an existing repo instead of a new one, that’s fine too; any subfolder works.

3. Add the pipeline’s files to that repo, keeping the folder structure (templates/, scripts/, and your connections.yaml) intact:

   • No git experience needed: on the Repos → Files screen, click ⋯ (top right) → Upload file(s), and repeat per folder (templates/rotate-connection-template.yaml, everything under scripts/, and your connections.yaml).

   • If you already use git: clone the repo, copy the unzipped package’s contents in, git add / git commit / git push.

2. Fill in your connections.yaml

1. Start from examples/connections.yaml in the downloaded package, or use the generator to produce one directly.

2. For each connection you need, add an entry to the connections list — see the connection type reference below. Use targetAppName to have the pipeline create a dedicated app registration + SP per connection, or targetAppId to pin an existing one.

3. connectionId is the Airflow conn_id and must be unique within the file — letters, digits, underscores, and hyphens only.

4. Pick a schedules: cron cadence shorter than the smallest validityDays among your connections, so every secret gets rotated before it expires.

3. Create the service connection

This is the one identity the pipeline runs as (see “One service connection” above) — its name must match the serviceConnectionName variable in your YAML file exactly.

Note

The Airflow application itself belongs to the platform and lives in the platform’s own tenant, not yours. Self-granting it the “Admin” role (below) only works once your organization has completed Entra ID onboarding — that’s what consents the platform’s Airflow application into your tenant, which is what your service connection’s SP needs to be able to assign it a role at all. If your Airflow instance is instead secured through the platform’s shared Keycloak realm, this rotation mechanism’s Airflow step doesn’t apply — talk to the platform team about rotating that connection’s secret.

1. Bottom-left corner of Azure DevOps, click Project Settings.

2. Under Pipelines, click Service connections.

3. Click New service connection.

4. Choose Azure Resource Manager, then click Next.

5. Choose Service principal (automatic) — Azure DevOps creates the app registration for you, so there’s no separate Entra ID step for this part.

6. Pick your Subscription (any subscription in your own tenant — this only matters for creating the service principal itself; which one you pick doesn’t affect its permissions, since everything this pipeline needs is granted at the Microsoft Entra ID level, not the subscription level). Leave Resource group empty (scope: Subscription).

7. Under Service connection name, enter exactly the same name as serviceConnectionName in your YAML file (e.g. customer-connections).

8. Uncheck Grant access permission to all pipelines. This service connection ends up holding sensitive rights (ownership of your target apps, the Graph permissions below) — don’t leave every pipeline in the project free to use it. Click Save. The first time you run the pipeline you create in the next section, Azure DevOps will show a banner asking to authorize that specific pipeline to use this connection — approve it there.

The service connection now exists, but it only has the base subscription role Azure DevOps gave it. It still needs two Microsoft Graph application permissions before the pipeline can do its job. Grant these once, as a Microsoft Entra ID administrator:

9. Open portal.azure.com, then Microsoft Entra ID → App registrations.

10. Find the app registration Azure DevOps just created — back on the service connection you made (Project Settings → Service connections → your connection → Manage Service Principal) opens it directly in the Entra portal.

11. Click API permissions → Add a permission → Microsoft Graph → Application permissions.

12. Search for and check both:

    • Application.ReadWrite.OwnedBy — needed if any connection uses targetAppName (lets the pipeline create that connection’s app registration).

    • AppRoleAssignment.ReadWrite.All — needed so the pipeline can self-grant itself the Airflow “Admin” role (see “Granting the Admin role” above).

    Click Add permissions.

13. Click Grant admin consent for <your organization>, then Yes to confirm. The permissions row should now show a green checkmark under Status.

If you don’t use targetAppName anywhere in your file, you can skip Application.ReadWrite.OwnedBy. If you’d rather not grant AppRoleAssignment.ReadWrite.All at all, skip it too — the pipeline will instead print manual-grant instructions the first time it runs (see “Granting the Admin role” above, option (a)).

4. Create the pipeline

1. Left sidebar, click Pipelines → New pipeline.

2. Where is your code? → Azure Repos Git → select the repo you pushed the files into.

3. Configure your pipeline → Existing Azure Pipelines YAML file.

4. Branch: main (or whichever branch you pushed to). Path: pick your connections.yaml file (e.g. /customer-connections.yaml).

5. Click Continue, review the YAML preview, then click the dropdown next to Run and choose Save (not Save and run — see the next step first).

6. Optional: rename the pipeline to something recognizable — open it, click ⋯ (top right) → Rename/move, e.g. “Rotate Airflow connections”.

5. Run it once, then trust the schedule

1. Open the pipeline and click Run pipeline → Run.

2. Watch each step. If the admin-role step fails, it prints the exact two fix options — see “Granting the Admin role” above and Troubleshooting below.

3. Once every step is green, your connections are live in Airflow — check the Airflow UI’s Admin → Connections page for the ``connectionId``(s) you configured.

4. Nothing else to do: the schedules: block in your YAML already arms recurring runs. Confirm the next run time under the pipeline’s Scheduled runs (clock icon, top right of the pipeline’s page).

Connection type reference

The pipeline template dispatches on connType. Types with a dedicated script take typed fields directly (no hand-written extraJson); any other connType falls through to the generic Set-AirflowConnection.ps1 step with hostName/extraJson. Login/password are always the rotated client_id/client_secret, and the tenant ID is derived at runtime from the service connection’s Azure context, never spelled out in YAML.

Type	connType	Typed fields	Notes
HTTP (generic OAuth-protected API)	http	hostName, extraJson (tenant_id, scope, token_url)	Airflow’s built-in HttpHook doesn’t do client-credentials OAuth itself — pair this with a small custom hook in your DAGs that reads these fields and performs the token exchange.
Databricks	databricks	hostName (workspace URL)	Matches apache-airflow-providers-databricks’s documented Azure AD SPN auth method (extra azure_tenant_id).
Fabric (FabricOperator)	fabric	delegatedScopes (User.Read Workspace.ReadWrite.All Item.ReadWrite.All Item.Execute.All)	Extra tenantId; workspace/item IDs are operator arguments, not connection fields. Scopes are declared on the target app’s API permissions by the rotate step — admin consent is a one-time manual step.
Fabric Capacity (FabricCapacityOperator)	fabric_capacity	—	Extra tenantId; subscription/resource group/capacity name are operator arguments. Targets the ARM management plane — assign the SP the custom role from “Fabric Capacity access” below.
Power BI	power_bi	—	Extra tenant_id; dataset/group IDs are operator arguments. See airflow-powerbi-provider.
Synapse (AzureSynapseHook)	azure_synapse	hostName (workspace URL), subscriptionId	Extras tenantId/subscriptionId. AzureSynapsePipelineHook reads the same fields, so one connection serves both.
Data Factory (AzureDataFactoryHook)	azure_data_factory	subscriptionId, resourceGroup, factoryName	Extras tenantId/subscriptionId plus resource_group_name/factory_name; resource group/factory are connection-level defaults operators can override.
Git (private DAGs repo)	git	repoUrl (HTTPS clone URL), trackingRef (branch)	Same dedicated-app-per-connection model — use targetAppName to give this connection its own SP, then see Git: authorizing the DAGs repo (one-time) below before expecting it to actually work. See also apache-airflow-providers-git-azure for the workload-identity alternative that needs no rotated secret at all.

Git: authorizing the DAGs repo (one-time)

Rotating the git connection’s secret only keeps its service principal’s credentials valid. It does not, by itself, give that service principal permission to read the DAGs repo. If the DAGs repo lives in Azure DevOps and you’re using targetAppName to give this connection its own dedicated service principal, do the following once per service principal, as an Azure DevOps organization administrator, before the pipeline’s first run against that connection:

1. Give the service principal an access level (a “license”). Azure DevOps requires one for every identity that touches it, service principals included — without this step, the repo permission you set up in step 2 exists but Git operations still fail.

   1. Organization Settings (gear icon, bottom-left) → Users → Add users.

   2. In the users field, paste the service principal’s Application (client) ID (found on the app registration’s Overview page in Microsoft Entra ID).

   3. Access level: choose Basic. Stakeholder does not include Git read access to a private repo, so it won’t work here.

   4. Add to project(s): pick the project the DAGs repo lives in. Click Add.

2. Authorize it on the specific repo — not organization-wide.

   1. Project Settings → Repositories → select the DAGs repo → Security tab.

   2. Click Add, and paste the same service principal identity.

   3. Set its permissions to Reader (covers Read / Code: Read — everything this connection type needs, since it only clones and pulls, never pushes).

Repeat both steps for every git connection entry that has its own dedicated service principal. If several git connections share one target app (via targetAppId instead), you only need to do this once for that shared identity.

Fabric Capacity access (one-time)

The fabric_capacity connection drives suspend/resume/scale through the ARM management plane, so its SP needs a role on the capacity resource, not Contributor. Create a minimal custom role once (Fabric Capacity (Smooth) Operator) granting Microsoft.Fabric/capacities/{read,write}, capacities/suspend/action, capacities/resume/action, and capacities/skus/read / skus/read, then assign it to the pipeline-created SP for that connection (run the pipeline once first so the SP exists). Leave out write if you only ever suspend/resume, not scale.

Troubleshooting

Symptom	Likely cause
Can’t reach the Airflow API (after retries)	airflowBaseUrl is wrong or the instance is down — not an auth problem, since this step retries with backoff first.
Admin-role step fails with manual-resolution steps	The service connection’s SP can’t self-grant the Airflow “Admin” role — grant it manually or give the SP AppRoleAssignment.ReadWrite.All, then re-run.
401/403 from the Airflow API even though the admin-role step passed	A freshly granted role can take a short while to propagate — re-run. If it persists, confirm the resolved Airflow app is the right one (pin it explicitly with airflowAppId if airflowAppName resolves ambiguously).
“does not exist” rotating the target secret	With targetAppId the target app must already exist — switch to targetAppName if you want the pipeline to create a dedicated app per connection.
Can’t create the target app	The service connection’s SP lacks app-creation rights — either have an admin create the app manually and pin it via targetAppId, or grant the SP Application.ReadWrite.OwnedBy and re-run.
Pipeline succeeded but the target system returns 401/403 to DAGs	A freshly created per-connection SP has credentials but no access yet — grant it the resource-side access once (workspace role, ARM RBAC, etc.).