AuthbridgeConfigReconciler overwrites per-namespace KEYCLOAK_REALM (regression in cdbed3b) — breaks non-default realms / token-exchange e2e

[pdettori]

Summary

AuthbridgeConfigReconciler unconditionally overwrites the KEYCLOAK_REALM key (and other Keycloak keys) of every watched namespace's authbridge-config ConfigMap with the platform-global realm on each reconcile. This silently discards any explicitly-set per-namespace realm, effectively removing the ability to run workloads against a non-default realm. It is a regression introduced by cdbed3b ("feat: operator-driven identity deployment").

Impact

Any namespace that needs a realm other than the platform default can no longer get one: even if an operator/admin/CI sets authbridge-config.KEYCLOAK_REALM=<other>, the reconciler resets it to Platform.KeycloakRealm (default "kagenti") within one reconcile cycle. The client-registration controller then registers the workload client in the wrong realm.

Concretely, this breaks the kagenti repo's token-exchange E2E suite (E2E K8s 1.35.0 (Kind) push workflow), which has been red continuously as a result. The first test TestClientCredentials fails with:

client_credentials failed: {"error":"invalid_client","error_description":"Invalid client or Invalid client credentials"}
assert 401 == 200

Root cause

On main, internal/controller/authbridgeconfig_controller.go:

// Update existing ConfigMap
...
for k, v := range desired {          // ~line 114 — overwrites existing keys
    cm.Data[k] = v
}

where desired is built from platform config:

realm := r.Platform.KeycloakRealm    // ~line 132
if realm == "" {
    realm = defaultKeycloakRealm     // "kagenti"  (line 28)
}
...
"KEYCLOAK_REALM": realm,             // ~line 159

The client-registration controller then reads that overwritten value:

KeycloakRealm: cm.Data["KEYCLOAK_REALM"],   // clientregistration_controller.go:357

So a workload in namespace tx-e2e (where the harness created realm tx-e2e and set KEYCLOAK_REALM=tx-e2e) gets its client registered in realm kagenti instead. A client_credentials grant against realm tx-e2e then returns invalid_client.

Reproduction

1. Build/run the operator from main.
2. Create a Keycloak realm tx-e2e.
3. In a watched namespace, kubectl apply an authbridge-config ConfigMap with KEYCLOAK_REALM: tx-e2e.
4. Observe (within ~one reconcile) that kubectl get cm authbridge-config -n <ns> -o jsonpath='{.data.KEYCLOAK_REALM}' reverts to kagenti.
5. The registered client (kagenti-keycloak-client-credentials-*, client-id.txt) lands in realm kagenti, not tx-e2e.

Version note

This regression is not present in v0.2.0-rc.6 — authbridgeconfig_controller.go does not exist there, so authbridge-config stays as created (e.g. Helm-managed) and per-namespace realms are honored. Verified on a live cluster running 0.2.0-rc.6: client_credentials for namespace clients returns HTTP 200. The breakage only appears when building the operator from main (which is what kagenti CI does via 20-build-operator.sh, default ref main).

Introduced by: cdbed3b — "feat: operator-driven identity deployment" (2026-06-05).

Proposed fix

Preserve an explicitly-set KEYCLOAK_REALM rather than clobbering it. Options:

1. Don't overwrite a key that's already present / non-empty in the existing CM (only fill in missing keys when reconciling an existing authbridge-config).
2. Honor a per-namespace override via a Namespace annotation/label (e.g. kagenti.io/keycloak-realm) that takes precedence over Platform.KeycloakRealm.
3. Only manage CMs the reconciler owns (skip CMs not labeled app.kubernetes.io/managed-by: kagenti-operator), so externally-provisioned config is left intact.

Option 1 or 2 restores per-namespace realm support without regressing the operator-driven defaulting added in cdbed3b.

Cross-refs

• Interim mitigation in kagenti: the token-exchange E2E step was made non-fatal (kagenti/kagenti PR #1982) so main regains a green/red signal while this is fixed.

[pdettori]

Verified on current upstream/main (fb22549) — proceeding with a fix

Re-checked the live code before fixing. Two notes since this issue was filed:

• The controller has moved into the nested module path: kagenti-operator/internal/controller/authbridgeconfig_controller.go (was internal/controller/...). The bug is unchanged.
• The reconciler now also stamps ownership labels on the existing CM (app.kubernetes.io/managed-by=kagenti-operator and kagenti.io/defaults=true) before clobbering the data. So authbridge-config is treated as a namespace-defaults source (consumed by defaults_config_reconciler.go / agentruntime_controller.go).

The offending update path:

cm.Labels["app.kubernetes.io/managed-by"] = "kagenti-operator"
cm.Labels[LabelNamespaceDefaults] = "true"   // kagenti.io/defaults
...
for k, v := range desired {
    cm.Data[k] = v          // ← unconditionally overwrites KEYCLOAK_REALM
}

Flow

flowchart LR
    H["harness: kubectl apply<br/>authbridge-config<br/>KEYCLOAK_REALM=tx-e2e"] --> R{Reconcile}
    R -->|"for k,v := range desired<br/>cm.Data[k]=v"| O["KEYCLOAK_REALM=kagenti<br/>(clobbered)"]
    O --> C["client-registration<br/>registers in realm 'kagenti'"]
    C --> X["client_credentials vs realm 'tx-e2e'<br/>→ invalid_client / 401"]

Loading

Chosen fix — Option 1 (preserve explicitly-set values)

authbridge-config is a defaults ConfigMap (kagenti.io/defaults=true), and cdbed3b introduced it purely as operator-driven defaulting. Defaults should fill gaps, not override explicit values. So on the update path the reconciler will only fill keys that are missing or empty, preserving any explicitly-set non-empty value (e.g. KEYCLOAK_REALM=tx-e2e):

for k, v := range desired {
    if existing, ok := cm.Data[k]; ok && existing != "" {
        continue // preserve explicitly-set value
    }
    cm.Data[k] = v
}

The create path (fresh kagenti-enabled namespace → full operator-provided defaults) and the label-stamping are unchanged, so the cdbed3b defaulting behavior is preserved for namespaces that have no config yet.

Tests

There is currently no test for this controller. Adding authbridgeconfig_controller_test.go (plain testing + fake client, matching clientregistration_controller_test.go) with:

1. Regression: existing CM KEYCLOAK_REALM=tx-e2e survives Reconcile (fails on current code).
2. Create path: no CM → created with full platform defaults + labels.
3. Missing-key fill: existing CM missing keys → operator fills only those.
4. Labels stamped on an existing (externally-created) CM.
5. Non-kagenti-enabled namespace → no CM created.

PR to follow.

Assisted-By: Claude Code