Skip to content

feat(proxy-init): add enforce-drop mode for proxy-sidecar egress enforcement #484

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
huang195 merged 4 commits into from
Jun 5, 2026
Merged

feat(proxy-init): add enforce-drop mode for proxy-sidecar egress enforcement #484

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
91708b7
feat(proxy-init): add enforce-drop mode for proxy-sidecar egress enfo…
huang195 Jun 5, 2026
17c05ec
fix(proxy-init): let established/related replies through enforce-drop
huang195 Jun 5, 2026
4f66a6b
test(proxy-init): enable strict mode in enforce-drop harness
huang195 Jun 5, 2026
f3101d7
docs(proxy-init): clarify CLUSTER_CIDRS portability and enforce-drop …
huang195 Jun 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 98 additions & 22 deletions authbridge/proxy-init/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# proxy-init

The `proxy-init` container sets up iptables rules so that traffic in
and out of an AuthBridge-injected pod is transparently redirected to
the AuthBridge sidecar's listeners. It runs once at pod startup as a
Kubernetes init container, then exits.
The `proxy-init` container programs iptables rules for an
AuthBridge-injected pod. It runs once at pod startup as a Kubernetes
init container, then exits. It has two modes, selected by the `MODE`
env var:

**`proxy-init` is only used in `envoy-sidecar` mode.** In
`proxy-sidecar` mode (the AuthBridge cluster default after
kagenti-operator#361) traffic interception is done via `HTTP_PROXY`
env vars on the workload container — no iptables, no init container.
| `MODE` | Used by | What it does |
|---|---|---|
| `redirect` (default) | `envoy-sidecar` | Transparently **REDIRECT**s pod traffic to the Envoy listeners. |
| `enforce-drop` | `proxy-sidecar` | Fail-closed egress guard — **DROP**s any egress that bypasses the forward proxy. |

## What it does
## `redirect` mode (envoy-sidecar)

`init-iptables.sh` writes iptables rules that:

Expand All @@ -28,21 +28,77 @@ env vars on the workload container — no iptables, no init container.
`INBOUND_PORTS_EXCLUDE` env vars (commonly used to exclude
Keycloak's port 8080 to avoid token-exchange loops).

## `enforce-drop` mode (proxy-sidecar)

In `proxy-sidecar` mode the workload is configured with `HTTP_PROXY`
pointing at AuthBridge's forward proxy. On its own that is purely
cooperative — an app that ignores `HTTP_PROXY` (or sets `NO_PROXY`)
egresses directly and bypasses AuthBridge. `enforce-drop` closes that
gap **without** transparently redirecting (you cannot REDIRECT raw
traffic into a CONNECT forward proxy): it installs a fail-closed guard
that DROPs any direct egress, forcing all external traffic through the
proxy regardless of whether the app honors `HTTP_PROXY`.

`init-iptables.sh` builds a dedicated `AB_EGRESS` chain hooked from
**`mangle` OUTPUT at position 1**, with this order:

1. `RETURN` ztunnel's own sockets (fwmark `0x539`) — keeps the mesh path working; a no-op when ambient is absent.
2. `RETURN` the proxy's own re-originated egress (`--uid-owner $PROXY_UID`, default 1337).
3. `RETURN` loopback (the app → proxy hop) and in-cluster CIDRs (`CLUSTER_CIDRS`, mesh/DNS).
4. `DROP` everything else — direct external egress, including UDP (QUIC/HTTP-3).

An IPv6 mirror drops external v6 egress (allowing loopback, link-local,
the proxy UID, and `CLUSTER_CIDRS6`).

> **`CLUSTER_CIDRS` is Kind-shaped by default.** The `10.0.0.0/8` default
> covers Kind (pods `10.244.0.0/16` + services `10.96.0.0/16`). Other
> distros differ — **OpenShift** uses services `172.30.0.0/16` and pods
> `10.128.0.0/14`, and `172.30.0.0/16` is **outside** `10/8`, so the
> default would drop in-cluster service traffic. On OCP/EKS/etc. you
> **must** override `CLUSTER_CIDRS` with the cluster's real pod+service
> ranges. The script logs the resolved value at startup, and the
> operator wiring (follow-up PR) sets it from the cluster's CIDRs.

> **`enforce-drop` intentionally ignores `OUTBOUND_PORTS_EXCLUDE`** (a
> `redirect`-mode knob). Any destination previously bypassed that way —
> e.g. a direct LLM endpoint at `host.docker.internal:11434` — is now
> dropped unless it goes through the forward proxy or falls within
> `CLUSTER_CIDRS`. That is the point: `enforce-drop` closes direct-egress
> holes. Operators relying on a bypass must route it through the proxy
> (or, for in-cluster targets, include it in `CLUSTER_CIDRS`).

**Why `mangle` OUTPUT, not `filter`:** when Istio ambient is active it
installs an in-pod `nat OUTPUT` REDIRECT (`ISTIO_OUTPUT` → ztunnel
`:15001`). The netfilter OUTPUT hook order is `raw → mangle → nat →
filter`, so a DROP in `mangle` evaluates the original destination and
fires **before** ambient's nat redirect can rewrite it; a DROP in
`filter` would run after nat and be defeated. `-I 1` also places the
chain ahead of Istio's appended (`-A`) mangle chain. This makes the
guard robust with no ambient, in-pod ambient, or node-level ambient.
See [`test-enforce-drop.sh`](./test-enforce-drop.sh), which proves the
preemption via packet counters.

## iptables backend

The script auto-detects `iptables-legacy` vs `iptables-nft` and uses
whichever the host kernel exposes. Override with `IPTABLES_CMD` if
needed.
whichever the host kernel exposes. Override with `IPTABLES_CMD` (and
`IP6TABLES_CMD`) if needed.

## Environment variables

| Variable | Default | Purpose |
|---|---|---|
| `PROXY_PORT` | `15123` | AuthBridge outbound listener port |
| `INBOUND_PROXY_PORT` | `15124` | AuthBridge inbound listener port |
| `PROXY_UID` | `1337` | UID of the AuthBridge sidecar process; excluded from outbound redirect |
| `OUTBOUND_PORTS_EXCLUDE` | (empty) | Comma-separated outbound port list to skip (e.g. `8080`) |
| `INBOUND_PORTS_EXCLUDE` | (empty) | Comma-separated inbound port list to skip |
| `POD_IP` | (required) | Set via Downward API; used as the DNAT target for ambient-mesh inbound |
| `IPTABLES_CMD` | auto-detected | Override iptables binary (`iptables-legacy` / `iptables-nft`) |
| Variable | Default | Mode | Purpose |
|---|---|---|---|
| `MODE` | `redirect` | both | `redirect` (envoy-sidecar) or `enforce-drop` (proxy-sidecar) |
| `PROXY_UID` | `1337` | both | UID of the AuthBridge sidecar process; exempted from redirect / drop |
| `PROXY_PORT` | `15123` | redirect | AuthBridge outbound listener port |
| `INBOUND_PROXY_PORT` | `15124` | redirect | AuthBridge inbound listener port |
| `OUTBOUND_PORTS_EXCLUDE` | (empty) | redirect | Comma-separated outbound port list to skip (e.g. `8080`) |

@mrsabath mrsabath Jun 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clarifying question: the OUTBOUND_PORTS_EXCLUDE / INBOUND_PORTS_EXCLUDE rows being redirect-only matches the implementation. I want to confirm the design intent: in enforce-drop, an operator who currently uses OUTBOUND_PORTS_EXCLUDE=11434 (Ollama) for redirect would lose direct egress to that port — they'd be forced through the forward proxy, which is exactly what enforce-drop exists to do. Just flagging because the docs/operator wiring (PR 2) might want to surface this as "if you've been relying on OUTBOUND_PORTS_EXCLUDE to bypass the proxy in proxy-sidecar mode, enforce-drop will close that off — that's the point."

@huang195 huang195 Jun 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — that's the intended design. enforce-drop deliberately does not honor OUTBOUND_PORTS_EXCLUDE: a destination previously bypassed via it (e.g. Ollama at host.docker.internal:11434) is now dropped unless it goes through the forward proxy or is in CLUSTER_CIDRS. Added a README note saying exactly that ("enforce-drop closes direct-egress holes"). Agreed the operator PR should call this out in its rollout notes.

| `INBOUND_PORTS_EXCLUDE` | (empty) | redirect | Comma-separated inbound port list to skip |
| `POD_IP` | (required in `redirect`) | redirect | Set via Downward API; DNAT target for ambient-mesh inbound. Not used by `enforce-drop`. |
| `CLUSTER_CIDRS` | `10.0.0.0/8` | enforce-drop | Comma-separated in-cluster CIDRs allowed direct (pods/services/DNS) |

@mrsabath mrsabath Jun 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: the 10.0.0.0/8 default covers Kind (pods 10.244/16 + services 10.96/16) cleanly, but the same operator dropped onto a stock OpenShift cluster will see services in 172.30.0.0/16 and pods in 10.128.0.0/14. Both are outside 10/8 for services. Could be worth a one-line callout in the README that the default is Kind-shaped and OCP/EKS users should override CLUSTER_CIDRS (or that init-iptables.sh could log the resolved value at startup — which it already does, just confirming that's the intended diagnostic path).

@huang195 huang195 Jun 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on OCP — 172.30.0.0/16 services are outside 10/8 and would be dropped. Added a README callout (f3101d7): the default is Kind-shaped and OCP/EKS must override CLUSTER_CIDRS, with the startup log as the diagnostic path. The follow-up operator PR will set CLUSTER_CIDRS from the cluster's real pod+service ranges, so the default only matters for standalone/Kind testing.

| `CLUSTER_CIDRS6` | (empty) | enforce-drop | IPv6 in-cluster CIDRs (dual-stack); empty drops all external v6 egress |
| `IPTABLES_CMD` | auto-detected | both | Override iptables binary (`iptables-legacy` / `iptables-nft`) |
| `IP6TABLES_CMD` | derived from `IPTABLES_CMD` | enforce-drop | Override ip6tables binary |

## Required Kubernetes capabilities

Expand All @@ -62,11 +118,31 @@ The image is published from CI as
`ghcr.io/kagenti/kagenti-extensions/proxy-init:<tag>` (build defined
in [`.github/workflows/build.yaml`](../../.github/workflows/build.yaml)).

## Testing

[`test-enforce-drop.sh`](./test-enforce-drop.sh) validates `enforce-drop`
mode in a private network namespace (`unshare --net`): it asserts the
`AB_EGRESS` rule structure and proves the `mangle` DROP preempts a
simulated Istio ambient `nat OUTPUT` REDIRECT via packet counters.
Requires root + iptables-nft on Linux (runs on CI; not macOS):

```sh
sudo ./test-enforce-drop.sh
```

## Where it gets injected

The kagenti-operator's mutating webhook injects the proxy-init
container automatically when the resolved AuthBridge mode is
`envoy-sidecar`. See
container automatically:

- `redirect` mode (`MODE` unset) when the resolved AuthBridge mode is
`envoy-sidecar`.
- `enforce-drop` mode (`MODE=enforce-drop`) when `proxy-sidecar`
egress enforcement is enabled (opt-in). _The operator wiring that
sets this lands in the follow-up kagenti-operator PR; this PR only
adds the mode to the image._

See
[`authbridge/demos/weather-agent/demo-ui-advanced.md`](../demos/weather-agent/demo-ui-advanced.md)
for an end-to-end demo and
[`authbridge/demos/token-exchange-routes/README.md`](../demos/token-exchange-routes/README.md)
Expand Down
134 changes: 132 additions & 2 deletions authbridge/proxy-init/init-iptables.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,22 @@

set -e

# --- Mode selection ---
# MODE selects the interception strategy:
# redirect (default) — envoy-sidecar: transparently REDIRECT pod traffic
# to the Envoy listeners (the behavior documented above).
# enforce-drop — proxy-sidecar: a fail-closed egress guard. The app is
# configured with HTTP_PROXY pointing at AuthBridge's forward
# proxy; this mode DROPs any egress that bypasses the proxy,
# forcing all external traffic through AuthBridge regardless of
# whether the app honors HTTP_PROXY. It installs no REDIRECT and
# no PREROUTING/inbound rules. See setup_enforce_drop() below.
MODE="${MODE:-redirect}"
case "${MODE}" in
redirect|enforce-drop) ;;
*) echo "ERROR: unknown MODE='${MODE}' (expected: redirect | enforce-drop)" >&2; exit 1 ;;
esac

# --- Auto-detect iptables backend ---
# Prefer iptables-legacy for maximum compatibility with Kubernetes networking.
# The nft backend sets rules in a different netfilter table that may not be
Expand Down Expand Up @@ -153,6 +169,17 @@ SSH_PORT="${SSH_PORT:-22}"
OUTBOUND_PORTS_EXCLUDE="${OUTBOUND_PORTS_EXCLUDE:-}"
INBOUND_PORTS_EXCLUDE="${INBOUND_PORTS_EXCLUDE:-}"

# enforce-drop mode: in-cluster destinations the agent may reach directly
# (pods / services / DNS) — everything else egressing the pod is dropped.
# Defaults to the RFC1918 10/8 block which covers typical Kind pod (10.244/16)
# and service (10.96/16) CIDRs; override with the cluster's actual ranges.
CLUSTER_CIDRS="${CLUSTER_CIDRS:-10.0.0.0/8}"
CLUSTER_CIDRS6="${CLUSTER_CIDRS6:-}" # IPv6 in-cluster CIDRs (dual-stack); empty = none

# IPv6 counterpart of the detected iptables backend (iptables-legacy ->
# ip6tables-legacy, iptables -> ip6tables). Override with IP6TABLES_CMD.
IP6T="${IP6TABLES_CMD:-$(echo "${IPT}" | sed 's/iptables/ip6tables/')}"

# Istio ztunnel defaults
ZTUNNEL_HBONE_PORT="${ZTUNNEL_HBONE_PORT:-15008}"
ZTUNNEL_MARK="${ZTUNNEL_MARK:-0x539/0xfff}" # 0x539 = 1337 decimal, ztunnel's socket fwmark
Expand All @@ -162,12 +189,115 @@ ISTIO_HEALTH_PROBE_SRC="${ISTIO_HEALTH_PROBE_SRC:-169.254.7.127}"
# It must be passed via the Kubernetes Downward API (status.podIP) or set manually.
# We use DNAT to the pod IP instead of REDIRECT to avoid needing route_localnet=1,
# which would require a privileged init container (to write to read-only /proc/sys).
if [ -z "${POD_IP}" ]; then
echo "ERROR: POD_IP environment variable is not set." >&2
# POD_IP is only needed by redirect mode (DNAT target for the ambient inbound
# rule). enforce-drop does no DNAT, so it does not require it.
if [ "${MODE}" = "redirect" ] && [ -z "${POD_IP}" ]; then
echo "ERROR: POD_IP environment variable is not set (required for redirect mode)." >&2
echo "Set it via the Kubernetes Downward API (status.podIP) or manually." >&2
exit 1
fi

# =============================================================================
# enforce-drop mode (proxy-sidecar fail-closed egress guard)
# =============================================================================
#
# proxy-sidecar configures the app with HTTP_PROXY=127.0.0.1:<forward-proxy>.
# Unlike redirect mode we do NOT transparently REDIRECT — you cannot redirect
# raw traffic into a CONNECT forward proxy. Instead we DROP any egress that
# leaves the pod without going through the proxy, forcing all external traffic
# through AuthBridge regardless of whether the app honors HTTP_PROXY.
#
# Placement — a dedicated chain hooked from *mangle* OUTPUT at position 1:
# * Istio ambient, when active, installs an in-pod `nat OUTPUT` REDIRECT
# (ISTIO_OUTPUT -> ztunnel :15001). The netfilter OUTPUT hook order is
# raw -> mangle -> nat -> filter, so a DROP in mangle evaluates the
# ORIGINAL destination and fires BEFORE ambient's nat redirect can rewrite
# it. A DROP in `filter` would run after nat and be defeated (dst already
# rewritten to 127.0.0.1). -I 1 also places us ahead of Istio's appended
# (-A) mangle ISTIO_OUTPUT chain.
# * Works identically with no ambient, in-pod ambient, or node-level ambient
# (in the node-level case our pod-netns rule runs before the packet ever
# reaches the host netns).
#
# Rule order in the chain: RETURN ztunnel's own sockets (fwmark 0x539, a no-op
# when ambient is absent) -> RETURN the proxy's own egress (PROXY_UID) ->
# RETURN loopback (app -> proxy) -> RETURN in-cluster CIDRs (mesh/DNS) ->
# DROP everything else (direct external egress, incl. UDP/QUIC).
setup_enforce_drop() {
CHAIN="AB_EGRESS"

echo "enforce-drop: installing fail-closed egress guard (mangle OUTPUT, chain ${CHAIN})"
echo "enforce-drop: exempt proxy UID=${PROXY_UID}; allowed in-cluster CIDRs=${CLUSTER_CIDRS}"

# --- IPv4 ---
${IPT} -t mangle -N "${CHAIN}" 2>/dev/null || true
${IPT} -t mangle -F "${CHAIN}"
# Replies to inbound connections (and related flows) are locally generated and
# also traverse OUTPUT — a reply is never a "bypass". Let established/related
# traffic through FIRST, so e.g. kubelet health-probe responses to an
# off-cluster node IP (in Kind the node is 172.18.0.0/16, outside CLUSTER_CIDRS)
# are not caught by the terminal DROP. Only NEW app-initiated flows are gated.
${IPT} -t mangle -A "${CHAIN}" -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
# ztunnel's own sockets (ambient) carry fwmark 0x539 — let them through so the
# mesh/HBONE path keeps working. No-op when ambient is not installed.
${IPT} -t mangle -A "${CHAIN}" -m mark --mark "${ZTUNNEL_MARK}" -j RETURN
# the AuthBridge proxy's own re-originated egress (must run as PROXY_UID).
${IPT} -t mangle -A "${CHAIN}" -m owner --uid-owner "${PROXY_UID}" -j RETURN
# app -> proxy over loopback (HTTP_PROXY target), and any loopback traffic.
${IPT} -t mangle -A "${CHAIN}" -o lo -j RETURN
${IPT} -t mangle -A "${CHAIN}" -d 127.0.0.0/8 -j RETURN
# in-cluster traffic (pods / services / DNS) — carried by the mesh, not the proxy.
for cidr in $(echo "${CLUSTER_CIDRS}" | tr ',' ' '); do
[ -n "${cidr}" ] && ${IPT} -t mangle -A "${CHAIN}" -d "${cidr}" -j RETURN
done
# everything else == direct external egress that bypassed the proxy. Drop it.
# No -p filter, so UDP (QUIC/HTTP-3) is dropped as well as TCP.
${IPT} -t mangle -A "${CHAIN}" -j DROP
# Hook at position 1 so we run before any appended Istio mangle chain and
# before nat OUTPUT.
if ! ${IPT} -t mangle -C OUTPUT -j "${CHAIN}" 2>/dev/null; then
${IPT} -t mangle -I OUTPUT 1 -j "${CHAIN}"
fi
echo "enforce-drop: IPv4 egress guard configured"

# --- IPv6 ---
# Cluster is IPv4-only by default; until v6 cluster CIDRs are wired
# (CLUSTER_CIDRS6), drop external v6 egress while allowing: established/related
# replies, loopback, link-local unicast (fe80::/10) and link-local multicast
# (ff02::/16, which carries NDP neighbor/router solicitations and MLD), the
# proxy UID, and ztunnel's mark.
if command -v "${IP6T%% *}" >/dev/null 2>&1 && ${IP6T} -t mangle -L >/dev/null 2>&1; then
${IP6T} -t mangle -N "${CHAIN}" 2>/dev/null || true
${IP6T} -t mangle -F "${CHAIN}"
${IP6T} -t mangle -A "${CHAIN}" -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
${IP6T} -t mangle -A "${CHAIN}" -m mark --mark "${ZTUNNEL_MARK}" -j RETURN
${IP6T} -t mangle -A "${CHAIN}" -m owner --uid-owner "${PROXY_UID}" -j RETURN
${IP6T} -t mangle -A "${CHAIN}" -o lo -j RETURN
${IP6T} -t mangle -A "${CHAIN}" -d ::1/128 -j RETURN
${IP6T} -t mangle -A "${CHAIN}" -d fe80::/10 -j RETURN
${IP6T} -t mangle -A "${CHAIN}" -d ff02::/16 -j RETURN
for cidr in $(echo "${CLUSTER_CIDRS6}" | tr ',' ' '); do
[ -n "${cidr}" ] && ${IP6T} -t mangle -A "${CHAIN}" -d "${cidr}" -j RETURN
done
${IP6T} -t mangle -A "${CHAIN}" -j DROP
if ! ${IP6T} -t mangle -C OUTPUT -j "${CHAIN}" 2>/dev/null; then
${IP6T} -t mangle -I OUTPUT 1 -j "${CHAIN}"
fi
echo "enforce-drop: IPv6 egress guard configured"
else
echo "enforce-drop: ip6tables unavailable — skipping IPv6 egress guard"
fi

echo "enforce-drop: fail-closed egress guard active"
}

# Dispatch enforce-drop here and exit; redirect mode falls through to the
# transparent-interception logic below.
if [ "${MODE}" = "enforce-drop" ]; then
setup_enforce_drop
exit 0
fi

# =============================================================================
# OUTBOUND traffic interception (nat OUTPUT)
# =============================================================================
Expand Down
Loading
Loading