Skip to content

Feat: Add egressEnforcement opt-out for proxy-sidecar mode#436

Merged
huang195 merged 3 commits into
rossoctl:mainfrom
akram:feat/egress-enforcement-optout
Jun 16, 2026
Merged

Feat: Add egressEnforcement opt-out for proxy-sidecar mode#436
huang195 merged 3 commits into
rossoctl:mainfrom
akram:feat/egress-enforcement-optout

Conversation

@akram

@akram akram commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Summary

On managed OpenShift platforms (ROSA HCP), iptables is unavailable in init containers due to missing kernel modules (iptable_nat) and SELinux restrictions (Red Hat KB 5990311). PR #407 made enforce-redirect proxy-init always-on in proxy-sidecar mode, which breaks pod creation on these platforms — pods are stuck in Init:CrashLoopBackOff.

This PR adds an egressEnforcement field to the AgentRuntime CR and namespace authbridge-runtime-config ConfigMap, allowing platforms without iptables to opt out of proxy-init injection.

Changes

Per-workload / per-namespace opt-out

  • AgentRuntime CRD: new egressEnforcement field with enum validation (enforce-redirect | none)
  • Resolution chain (same pattern as authBridgeMode and mtlsMode):
    1. AgentRuntime CR .spec.egressEnforcement (per-workload)
    2. Namespace authbridge-runtime-config ConfigMap egressEnforcement field
    3. enforce-redirect (cluster default — preserves current behavior)
  • Unknown values fail closed to enforce-redirect
  • envoy-sidecar mode unaffected — its proxy-init (redirect mode) is structural

Platform-level governance

  • proxy.allowedEgressEnforcement in platform config (Helm values → kagenti-platform-config): restricts which modes workloads may select
    • ["enforce-redirect"] — no opt-out allowed
    • ["none"] — iptables disabled cluster-wide (ROSA HCP)
    • ["enforce-redirect", "none"] — workloads choose (default)
  • Webhook overrides resolved value to the first allowed value when not permitted

Security model when egressEnforcement: none

  • Cooperative agents (respect HTTP_PROXY) → authenticated through authbridge ✓
  • Non-cooperative agents → egress without token → destination's inbound AuthBridge rejects → agent breaks (acceptable)
  • Data exfiltration → mitigated by egress NetworkPolicy restricting destinations

Files changed

  • api/v1alpha1/agentruntime_types.go — CRD field
  • internal/webhook/injector/constants.go — enforcement mode constants
  • internal/webhook/injector/agentruntime_config.go — CR extraction
  • internal/webhook/injector/namespace_config.goExtractEgressEnforcement()
  • internal/webhook/injector/pod_mutator.go — resolution chain + conditional injection + platform policy
  • internal/webhook/config/types.goAllowedEgressEnforcement field + validation
  • internal/webhook/config/defaults.go — default allows both modes
  • internal/webhook/injector/pod_mutator_test.go — 10 tests

Test plan

  • Unit tests: 10 new tests covering default, none, enforce-redirect, namespace ConfigMap, CR override, unknown value fail-closed, envoy-sidecar ignore, platform policy (block/allow/only-none)
  • go build ./... passes
  • Deployed on ROSA HCP cluster with egressEnforcement: none — pods start without proxy-init, authbridge-proxy sidecar injected, no iptables crash
  • Full e2e with client registration (blocked by unrelated upstream issue with ClientRegistrationReconciler)

Context

  • Discussion: #kagenti Slack thread (2026-06-15)
  • Jira: RHAIENG-5702
  • Related: kagenti-extensions#502 (iptables detection bug)

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

@akram
akram requested a review from a team as a code owner June 16, 2026 09:42
@akram
akram force-pushed the feat/egress-enforcement-optout branch 2 times, most recently from a1693ad to 07ec6f7 Compare June 16, 2026 11:59

@huang195 huang195 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well-structured, security-conscious feature with a correct fail-closed resolution chain (CR > namespace > enforce-redirect default; unknown values fail closed; envoy-sidecar correctly unaffected) and strong unit coverage. One blocking issue: the CRD manifests were not regenerated, so the marquee per-workload .spec.egressEnforcement field is pruned by the API server and silently no-ops on a real cluster — only the namespace-ConfigMap and platform-policy paths actually work. Details inline. Two design suggestions and one nit round it out.

Areas reviewed: Go (CRD type, webhook resolution + injection, platform policy), tests, CRD manifests, Helm values
Author: akram (MEMBER — maintainer)
Commits: 2, both signed-off (DCO passing)
CI: Build / Unit / Integration / Lint / CodeQL / Trivy green; E2E pending (PR notes full e2e blocked by an unrelated ClientRegistrationReconciler issue — so the CRD-pruning bug isn't caught by automated e2e either).

Once make manifests is committed (and a quick check that kubectl apply of an AgentRuntime preserves egressEnforcement), this is an easy approve.

Assisted-By: Claude Code

Comment thread kagenti-operator/api/v1alpha1/agentruntime_types.go
Comment thread kagenti-operator/internal/webhook/config/types.go
Comment thread kagenti-operator/internal/webhook/injector/pod_mutator.go Outdated
Comment thread kagenti-operator/internal/webhook/injector/pod_mutator.go Outdated
akram added 3 commits June 16, 2026 15:45
On managed OpenShift platforms (ROSA HCP), iptables is unavailable in
init containers due to missing kernel modules (iptable_nat) and SELinux
restrictions. PR rossoctl#407 made enforce-redirect always-on in proxy-sidecar
mode, which breaks pod creation on these platforms.

Add an egressEnforcement field (enforce-redirect | none) to the
AgentRuntime CR and namespace authbridge-runtime-config ConfigMap,
following the same resolution chain as authBridgeMode and mtlsMode:

  1. AgentRuntime CR .spec.egressEnforcement (per-workload)
  2. Namespace authbridge-runtime-config egressEnforcement (namespace)
  3. "enforce-redirect" (cluster default, preserves current behavior)

When set to "none", proxy-init is not injected in proxy-sidecar/lite
modes. Egress relies on HTTP_PROXY (cooperative) + inbound AuthBridge
on destinations + NetworkPolicy. Unknown values fail closed to
enforce-redirect.

envoy-sidecar mode is unaffected — its proxy-init (redirect mode) is
structural and always injected regardless of this setting.

Ref: RHAIENG-5702
Ref: rossoctl/cortex#502

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Akram <akram.benaissi@gmail.com>
Adds proxy.allowedEgressEnforcement to the platform config (Helm
values → kagenti-platform-config ConfigMap). The webhook validates
the resolved egressEnforcement value against this list and overrides
it to the first allowed value when not permitted.

This gives platform admins governance over egress enforcement modes:
- ["enforce-redirect"]           → no opt-out allowed
- ["none"]                       → iptables disabled cluster-wide (ROSA HCP)
- ["enforce-redirect", "none"]   → workloads choose (default)

Default allows both modes for backward compatibility.

Ref: RHAIENG-5702

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Akram <akram.benaissi@gmail.com>
- Regenerate CRD manifests (make manifests) — egressEnforcement field
  now appears in agent.kagenti.dev_agentruntimes.yaml with enum
  validation
- Replace stringInSlice helper with slices.Contains (stdlib)
- Reject empty allowedEgressEnforcement in Validate() — empty list
  was a footgun (bypass governance); must be explicitly set
- Add allowedEgressEnforcement to charts/kagenti-operator/values.yaml
  with doc comment for discoverability

Ref: RHAIENG-5702

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Akram <akram.benaissi@gmail.com>
@akram
akram force-pushed the feat/egress-enforcement-optout branch from 305021e to ecea178 Compare June 16, 2026 13:50

@huang195 huang195 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review after the rebase (HEAD ecea178). All feedback from both prior rounds is resolved:

  • CRD regeneratedegressEnforcement is now present with enum: [enforce-redirect, none] in both charts/...agentruntimes.yaml and config/crd/bases/...agentruntimes.yaml, so the per-workload .spec.egressEnforcement field is no longer pruned by the API server. This was the round-1 blocker.
  • gofmt fixed — the leftover double blank line from removing stringInSlice is gone; Lint is green. (round-2 blocker)
  • allowedEgressEnforcement surfaced and documented in charts/kagenti-operator/values.yaml.
  • Empty-list hardenedValidate() now rejects an empty allowedEgressEnforcement with an actionable error, so the allowed[0] override is safe.
  • slices.Contains replaces the hand-rolled helper.

Clean implementation overall: secure-by-default (enforce-redirect), fail-closed on unknown values, envoy-sidecar correctly unaffected, good unit coverage (10 tests).

Author: akram (MEMBER — maintainer)
CI: Build / Unit / Integration / Lint / Helm / YAML green; E2E pending (PR notes full e2e is blocked by an unrelated ClientRegistrationReconciler issue).
Agent/IDE config (.claude/.vscode): none

LGTM.

Assisted-By: Claude Code

@huang195
huang195 merged commit 060dfbf into rossoctl:main Jun 16, 2026
16 checks passed
varshaprasad96 added a commit to varshaprasad96/kagenti-operator that referenced this pull request Jun 18, 2026
Add missing workloadKind parameter to 10 InjectAuthBridge test calls
introduced by upstream PRs (rossoctl#436 egress enforcement, rossoctl#434 namespace
realm, rossoctl#432 iptables backend) that merged after this branch diverged.

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants