Accuracy note (2026-07-29 audit): This post was reviewed against current official documentation in July 2026 and contains inaccuracies relative to the current state of the tools described. The post is retained for its workflow reasoning; the specific factual issues are:

Two refinements: (1) Current Kafka supports standards-based OAuth/OIDC token acquisition and validation using built-in handlers and properties (sasl.oauthbearer.token.endpoint.url, client credentials in JAAS, sasl.oauthbearer.jwks.endpoint.url, issuer, audience); a custom AuthenticateCallbackHandler is not universally required — reserve custom callbacks for unsupported token-acquisition paths. (2) When the minimum refresh period plus buffer exceeds remaining token lifetime, Kafka ignores those two constraints; this does not itself mean refresh ‘may never’ be scheduled.

Reference docs to verify against:

  • Apache Kafka — SASL/OAUTHBEARER — https://kafka.apache.org/documentation/#security_sasl

When a Spark structured-streaming or batch job talks to a Kafka cluster secured with OAuth, a failure in the auth handshake surfaces as a vague SASL authentication failed or a stream that silently retries and never makes progress. Unlike a plain SASL/PLAIN failure (wrong password, obvious), OAuth failures sit at the intersection of three systems — the Kafka client, the OAuth authorization server, and the token-refresh logic inside the client — and the error message rarely tells you which one is at fault.

This post covers how the Kafka SASL/OAUTHBEARER handshake actually works, the failure modes you will hit in a Spark pipeline, and the specific configuration that fixes each one. Every config key below is a real Kafka client property, cited to the official Apache Kafka documentation.

Timeline of the Kafka SASL/OAUTHBEARER handshake across Spark client, auth server, and Kafka broker, with the three failure points: token acquisition, broker rejection, and refresh failure.
The OAuth handshake has three failure surfaces: getting the token, the broker accepting it, and the refresh thread renewing it before expiry. The error message rarely tells you which one.

How Kafka OAuth works

Kafka’s OAuth support uses the OAUTHBEARER SASL mechanism. The Kafka documentation describes it under “Authentication using SASL”. On the client side, you set three things:

  1. sasl.mechanism=OAUTHBEARER — tells the Kafka client to use the OAuth mechanism. The default SASL mechanism is GSSAPI, so this must be set explicitly.
  2. security.protocol=SASL_SSL (or SASL_PLAINTEXT for non-production) — the transport that carries SASL. Production Kafka should always be SASL_SSL.
  3. sasl.jaas.config — a JAAS login configuration that names the login module and its options. For the built-in unsecured mode, it looks like:
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
    unsecuredLoginStringClaim_sub="alice";

The OAuthBearerLoginModule is the class Kafka provides. For production, the Kafka docs are explicit: the built-in unsecured token creation and validation are “suitable only for non-production use.” You must register a custom AuthenticateCallbackHandler implementation that acquires a real OAuth token from your authorization server (Azure AD / Entra ID, Okta, Auth0, etc.) and handles OAuthBearerTokenCallback. This is done with sasl.login.callback.handler.class:

sasl.login.callback.handler.class=com.example.CustomOAuthLoginCallbackHandler

The token is a JWT. The Kafka docs note that under the default principal builder, the principalName of the OAuthBearerToken becomes the authenticated Kafka Principal used for ACLs.

The token-refresh problem

The most common OAuth failure in a long-running Spark streaming job is not the initial handshake — it is the refresh. OAuth tokens expire (typically in an hour), and the Kafka client has a background login-refresh thread whose job is to renew the token before it expires. If the refresh fails — the authorization server is unreachable, the refresh hits a rate limit, or the refresh window is misconfigured — the next time Kafka tries to authenticate a new connection (a reconnect after a broker bounce, a new consumer joining), it fails with an auth error and the stream stalls.

The Kafka documentation defines four knobs that govern refresh behavior (these appear in the producer, consumer, and broker config docs):

  • sasl.login.refresh.window.factor (default 0.8) — the refresh thread waits until this fraction of the token’s lifetime has elapsed before trying to refresh. Range 0.5–1.0.
  • sasl.login.refresh.window.jitter (default 0.05) — random jitter added to the refresh sleep, relative to the token lifetime. Range 0.0–0.25.
  • sasl.login.refresh.min.period.seconds (default 60) — the minimum time the refresh thread will wait between refresh attempts. Range 0–900.
  • sasl.login.refresh.buffer.seconds (default 300) — the buffer the client tries to maintain before token expiry. Range 0–3600.

The docs note that sasl.login.refresh.min.period.seconds and sasl.login.refresh.buffer.seconds are ignored together if their sum exceeds the token’s remaining lifetime. This is the trap: if your authorization server issues short-lived tokens (say, 10 minutes) and you leave sasl.login.refresh.buffer.seconds at its default 300 (5 minutes), the buffer alone is half the token’s life — the refresh math breaks and the client may never successfully schedule a refresh, so every reconnect re-authenticates with an expiring or expired token.

Fix: Set sasl.login.refresh.buffer.seconds to a small fraction of your token’s actual lifetime (e.g., 60 for a 10-minute token), and lower sasl.login.refresh.min.period.seconds if your tokens are short-lived. Verify the values against the token’s exp claim.

Failure mode 1: token acquisition fails at startup

If the very first connection fails, the problem is token acquisition, not refresh. The custom callback handler in sasl.login.callback.handler.class either cannot reach the authorization server, has the wrong client credentials, or is requesting the wrong audience/scope.

On Azure Synapse, the authorization server is usually Azure AD / Entra ID. The Synapse TokenLibrary docs document how Synapse Spark acquires AAD tokens for linked services: mssparkutils.credentials.getToken() returns a bearer token for a given resource, and linked-service-based token providers (LinkedServiceBasedTokenProvider) are configured with fs.azure.account.oauth.provider.type.<storage-account> and the client-id/client-secret/client-endpoint properties. However, the TokenLibrary docs cover storage and Key Vault scenarios; there is no Synapse-specific documented path for wiring Synapse-managed identity into a Kafka sasl.login.callback.handler.class.

So for a Spark-to-Kafka OAuth pipeline on Synapse, the custom callback handler is the integration point you own. If you are using managed identity or a service principal to get the AAD token, the handler must call the AAD token endpoint, receive the JWT, and hand it to Kafka’s OAuthBearerTokenCallback. If that handler logs nothing and returns null, the Kafka client fails auth with no useful message.

Fix: Enable logging in your custom callback handler (it runs inside the executor JVM — logs go to the executor stdout/stderr that Synapse Monitor exposes). Log the token endpoint URL, the HTTP status, and the response body on failure. Confirm the aud claim in the returned token matches what the Kafka broker expects.

Failure mode 2: the broker rejects the token

The Kafka client acquires a valid token, but the broker rejects it. This happens when the broker-side validation disagrees with the client. The Kafka docs document the broker-side validation options for the unsecured validator (unsecuredValidatorPrincipalClaimName, unsecuredValidatorScopeClaimName, unsecuredValidatorRequiredScope, unsecuredValidatorAllowableClockSkewMs), but in production the broker runs a custom AuthenticateCallbackHandler registered via listener.name.sasl_ssl.oauthbearer.sasl.server.callback.handler.class (the docs note it must be prefixed with listener and mechanism names). The broker’s validator checks the token’s signature, its exp, its iat, and its required scopes.

Common causes:

  • Clock skew between the Spark executor and the authorization server. The unsecured validator has unsecuredValidatorAllowableClockSkewMs (default 0 — zero tolerance); a production validator usually has a similar knob.
  • Wrong scope. The token does not contain the scope the broker requires (unsecuredValidatorRequiredScope).
  • Wrong principal claim. The broker looks for sub (the default) but the authorization server puts the principal in a different claim.

Fix: If you control the broker, check the broker-side validator logs. If you do not, inspect the token the client is sending — decode the JWT and compare its claims against the broker’s documented requirements. Fix clock skew at the VM level (Synapse executors should be NTP-synced, but a drifted node can still produce this).

Failure mode 3: connection/retry storms

A transient auth failure (a broker bounce, a brief authorization server outage) triggers Kafka’s reconnect and retry logic. The Kafka docs document the connection-level backoff: reconnect.backoff.ms (default 50), reconnect.backoff.max.ms (default 1000), retry.backoff.ms (default 100), and retry.backoff.max.ms (default 1000). These back off exponentially with 20% jitter.

The important subtlety: these retry knobs govern retriable failures — connection drops, transient broker errors. They do not make an invalid or expired token retriable. If the token is the problem, more retries just produce more auth failures faster, which can look like a storm of SASL authentication failed messages and, on some authorization servers, get your client rate-limited or temporarily blocked.

Fix: Distinguish the two in your logs. If the error is SASL authentication failed: invalid token or an expired-token message, the fix is token refresh (failure mode 1), not connection retries. Only tune retry.backoff.ms / reconnect.backoff.ms if the error is a connection-level failure (connection refused, timeout) that happens between successful auths.

Putting it together

A Spark-to-Kafka OAuth pipeline fails for one of three reasons: the client cannot get a token (callback handler or authorization server), the token it gets is rejected by the broker (scope, signature, clock skew), or the token expires and the refresh thread cannot renew it in time (refresh window misconfigured against the token lifetime). The error message alone rarely tells you which. The fix is to instrument the custom callback handler, decode the JWT the client is sending, and match the refresh configuration (sasl.login.refresh.*) to the actual token lifetime. OAuth is not magic — it is a stateful handshake with an expiration clock, and the failure modes all reduce to the clock or the claims being wrong.

Sources