The cause: the job's credential was quietly deleted weeks ago, because it hadn't been used in 90 days.

Since 16 September 2026, that's how Azure Databricks works. It's a good security change, but it breaks exactly the jobs you think about least.

Service principals and secrets, in plain language

A service principal is a robot account. People log in as themselves, but a scheduled job, a Terraform pipeline or a GitHub Actions workflow needs its own identity. That's the service principal.

An OAuth secret is the robot's password. The robot shows its ID and secret to Databricks and gets back an access token valid for one hour. When the token expires, it asks for a new one.

Think of a contractor's office badge. The badge gets them through the gate, where they collect a day pass to work inside. The badge is the secret; the day pass is the token.

What changed

Azure Databricks now automatically deletes a service principal's OAuth secret after 90 days without use. It already did the same for unused personal access tokens; OAuth secrets now follow the same rule.

In badge terms: if a contractor's badge isn't swiped for 90 days, security deactivates it.

Two details that trip people up:

• This is separate from the secret's expiry date. You set a lifetime of up to 2 years when you create a secret. Even a 2-year secret is deleted if it sits unused for 90 days.
• The failure doesn't explain itself. Nothing breaks on the day the secret is deleted. The job fails weeks later, on its next run, with an authentication error — so teams often check permissions and networking before thinking of a missing secret.

Who gets hit

Daily and weekly jobs use their secret constantly, so they never reach 90 days. The risk is the automation that runs rarely:

• Quarterly or yearly jobs, like quarter-end finance reports. Ninety-one days between runs is enough to lose the secret.
• Disaster recovery pipelines. Built, tested once, then left alone until an emergency — the worst moment to find a dead credential.
• Terraform pipelines for stable infrastructure. If nobody changes that workspace for three months, its pipeline's secret disappears.
• Old test or migration setups that someone revives later.

A quarterly job loses its secret before its next run

Illustrative: a job that runs every quarter-end, roughly every 91 days

How to protect yourself

1. Find the rare-use automation

List every service principal behind quarterly, yearly, disaster recovery or rarely changed Terraform jobs. Anything that authenticates less often than every 90 days is on your risk list.

2. Go secretless where you can (best fix)

A secret that doesn't exist can't be deleted.

• GitHub Actions: use workload identity federation. GitHub proves the workflow's identity directly to Databricks through a federation policy. There's no secret to store in GitHub, rotate or lose.
• Azure-hosted automation: use Microsoft Entra service principals or managed identities, so Azure handles the credential.

3. If you must keep secrets

• Store them in Azure Key Vault, never in code or a pipeline variable.
• Give each one an owner and a documented rotation date.
• Add a "can we log in?" check to your DR drills and quarterly runbooks, a week before the real run.
• Remember a service principal can hold up to five secrets, and each is shown only once when created.

4. Know the recovery

If it does happen, the fix is simple: generate a new secret for the service principal and update it wherever it's stored. The hard part is recognising the cause — so when a rarely run job suddenly fails to authenticate, check the secret first.

The bottom line

Deleting forgotten credentials is the right call for security. But it shifts the risk onto the jobs you think about least. Treat it as a nudge: find your rare-use automation now, and move it to secretless authentication before the next quarter-end does it for you.

Do you know which of your pipelines run less than once every 90 days? Tell me in the comments.

=========================================================

Muqtadirmalik Quazi (Malik) is a Platform Engineer who administers Azure Databricks at enterprise scale. He writes about Azure Databricks administration, AI governance and platform engineering for data and AI. Connect with Muqtadirmalik Quazi on LinkedIn, or follow @m.malikquazi on Medium and X.

=========================================================

References

• Azure Databricks September 2026 release notes (Microsoft Learn)
• Authorize service principal access with OAuth M2M (Microsoft Learn)
• Configure a federation policy (Microsoft Learn)

Databricks’ own release note carries the warning admins should read twice: this can increase model traffic, including to external providers like OpenAI or Anthropic. External traffic means a second invoice you don’t see inside Databricks.

Think of it like a company handing fuel cards to every employee. That’s great for productivity, but only if the card system has per-person limits, a monthly budget and receipts. In Azure Databricks, that card system is Unity Gateway.

What Unity Gateway is

Unity Gateway is Databricks’ control plane for AI traffic, built on Unity Catalog. Every call from a user, a job or an AI agent to a model or tool goes through it. It governs three kinds of objects:

• Model services — Databricks-hosted foundation models in system.ai, such as GPT, Claude and Gemini, billed pay-per-token.
• Model provider services — your own OpenAI or Anthropic accounts, connected with your own keys (BYOK).
• MCP services — the tools AI agents call.

The admin-friendly part: you grant access to these with the same Unity Catalog privileges you already use for tables.

How AI traffic flows through Unity Gateway in Azure Databricks:

The four controls every admin should know

1. Permissions: who can use which model Grant or revoke access to model services, provider services and MCP services like any Unity Catalog object. You can also restrict which Databricks-hosted foundation models the whole organisation, or a single group, can use. Start here: not every team needs the most expensive model.
2. Rate limits: how fast Rate limits cap traffic in requests per minute (QPM) and, for model services, tokens per minute (TPM). You set them at three levels:

Service — a ceiling for all traffic. Hit it and everyone is blocked.

User (default) — the limit every user gets.

Custom — overrides for specific users, service principals or groups.

Things worth knowing: there are no limits by default. Blocked callers get an HTTP 429, so apps need retry with backoff. Short bursts above the limit can slip through. And you get a maximum of 20 limits per service, only 5 of them group-based.

3. Budgets how much: Budgets scoped to Unity Gateway track monthly spend in near real time. You can set a shared threshold for everyone, a per-user threshold, and overrides for power users. At each threshold you choose to send an alert or block usage until the budget resets.

Two caveats: blocking is approximate, so spend can run slightly over before it kicks in. And budgets don’t cover provisioned throughput or traffic that bypasses the gateway. External provider spend can be included, but that option is still in Beta.

4. Visibility: what actually happened

• Usage system tables show who called which model, tokens used and latency. Use them for cost attribution and chargeback.
• Inference tables log the full request and response of a model service into a Unity Catalog Delta table. Great for debugging and audit but they’re a billed feature, rows can take up to an hour to arrive, and payloads above 10 MiB aren’t logged.

Week-one admin checklist

• List who has the consumer access entitlement today.
• Restrict expensive models to the groups that need them.
• Set a default per-user rate limit on each model service, plus a service-wide ceiling.
• Create a Unity Gateway budget with an alert at 80% and a block at 100%.
• Decide whether external provider spend belongs in the budget.
• Build a weekly chargeback view from the usage tables.
• Turn on inference tables only where you truly need full payloads.

My honest concerns as an operator

• Inference tables store raw prompts. If users paste customer data into a chat, it now lives in a Delta table. Lock down that schema like any table holding sensitive data.
• Budgets are not a hard financial cap. Databricks itself says not to treat them as an absolute spend limit. Your external provider invoice remains the real number.
• 429s look like outages to users. A tight rate limit without retry logic in apps becomes a support ticket. Tell teams before you enforce limits.
• Default is wide open. No rate limits are set out of the box. If you do nothing, your first control is the monthly bill.

The bottom line

AI in the data platform is no longer a data science side project; it’s a shared utility, like compute. And like compute, it needs a platform admin to decide who gets it, how much and at what cost. Unity Gateway gives you those levers — but none of them are switched on by default.

Have you set rate limits or budgets on your AI models yet, or are you waiting for the first surprise bill? Tell me in the comments.

Muqtadir Malik Quazi is a Platform Engineer who administers Azure Databricks at enterprise scale. He writes about Azure Databricks administration, AI governance and platform engineering for data and AI. Connect with Muqtadir Malik Quazi on LinkedIn: https://www.linkedin.com/in/muqtadirmalik, or follow @m.malikquazi on Medium and X.

References

• AI governance with Unity Gateway (Microsoft Learn)
• Apply rate limits to model and MCP services (Microsoft Learn)
• Manage budgets for Unity Gateway (Microsoft Learn)
• Log requests and responses to inference tables (Microsoft Learn)
• Azure Databricks October 2026 release notes (Microsoft Learn)