From 91708b75b5df9262268d6370995e20bc2f7f41fd Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Fri, 5 Jun 2026 13:20:23 -0400 Subject: [PATCH 1/4] feat(proxy-init): add enforce-drop mode for proxy-sidecar egress enforcement proxy-sidecar mode routes outbound traffic through AuthBridge via the HTTP_PROXY env var, which is purely cooperative: a workload that ignores HTTP_PROXY (or sets NO_PROXY) egresses directly and bypasses AuthBridge entirely. Add a fail-closed egress guard so all external traffic is forced through the forward proxy regardless of whether the workload honors HTTP_PROXY. init-iptables.sh gains a MODE switch. Default `redirect` preserves the existing envoy-sidecar behavior byte-for-byte. MODE=enforce-drop builds an AB_EGRESS chain hooked from mangle OUTPUT at position 1: - RETURN ztunnel sockets (fwmark 0x539), the proxy UID, loopback, and the in-cluster CIDRs (pods/services/DNS) - DROP everything else (direct external egress, including UDP/QUIC) plus an IPv6 mirror that drops external v6 egress. Placement is the mangle table, not filter: when Istio ambient is active it installs an in-pod `nat OUTPUT` REDIRECT (ISTIO_OUTPUT -> ztunnel :15001). The netfilter OUTPUT hook order is raw -> mangle -> nat -> filter, so a DROP in mangle evaluates the original destination and fires before ambient's nat redirect can rewrite it; a DROP in filter would run after nat and be defeated. -I 1 also keeps us ahead of Istio's appended mangle chain. This makes the guard robust with no ambient, in-pod ambient, or node-level ambient. test-enforce-drop.sh validates the rule structure and proves the mangle DROP preempts a simulated ISTIO_OUTPUT nat REDIRECT via packet counters, in an unshare --net namespace. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- authbridge/proxy-init/README.md | 102 +++++++++++++---- authbridge/proxy-init/init-iptables.sh | 124 ++++++++++++++++++++- authbridge/proxy-init/test-enforce-drop.sh | 87 +++++++++++++++ 3 files changed, 289 insertions(+), 24 deletions(-) create mode 100755 authbridge/proxy-init/test-enforce-drop.sh diff --git a/authbridge/proxy-init/README.md b/authbridge/proxy-init/README.md index eeb263f15..84025390c 100644 --- a/authbridge/proxy-init/README.md +++ b/authbridge/proxy-init/README.md @@ -1,16 +1,16 @@ # proxy-init -The `proxy-init` container sets up iptables rules so that traffic in -and out of an AuthBridge-injected pod is transparently redirected to -the AuthBridge sidecar's listeners. It runs once at pod startup as a -Kubernetes init container, then exits. +The `proxy-init` container programs iptables rules for an +AuthBridge-injected pod. It runs once at pod startup as a Kubernetes +init container, then exits. It has two modes, selected by the `MODE` +env var: -**`proxy-init` is only used in `envoy-sidecar` mode.** In -`proxy-sidecar` mode (the AuthBridge cluster default after -kagenti-operator#361) traffic interception is done via `HTTP_PROXY` -env vars on the workload container — no iptables, no init container. +| `MODE` | Used by | What it does | +|---|---|---| +| `redirect` (default) | `envoy-sidecar` | Transparently **REDIRECT**s pod traffic to the Envoy listeners. | +| `enforce-drop` | `proxy-sidecar` | Fail-closed egress guard — **DROP**s any egress that bypasses the forward proxy. | -## What it does +## `redirect` mode (envoy-sidecar) `init-iptables.sh` writes iptables rules that: @@ -28,21 +28,60 @@ env vars on the workload container — no iptables, no init container. `INBOUND_PORTS_EXCLUDE` env vars (commonly used to exclude Keycloak's port 8080 to avoid token-exchange loops). +## `enforce-drop` mode (proxy-sidecar) + +In `proxy-sidecar` mode the workload is configured with `HTTP_PROXY` +pointing at AuthBridge's forward proxy. On its own that is purely +cooperative — an app that ignores `HTTP_PROXY` (or sets `NO_PROXY`) +egresses directly and bypasses AuthBridge. `enforce-drop` closes that +gap **without** transparently redirecting (you cannot REDIRECT raw +traffic into a CONNECT forward proxy): it installs a fail-closed guard +that DROPs any direct egress, forcing all external traffic through the +proxy regardless of whether the app honors `HTTP_PROXY`. + +`init-iptables.sh` builds a dedicated `AB_EGRESS` chain hooked from +**`mangle` OUTPUT at position 1**, with this order: + +1. `RETURN` ztunnel's own sockets (fwmark `0x539`) — keeps the mesh path working; a no-op when ambient is absent. +2. `RETURN` the proxy's own re-originated egress (`--uid-owner $PROXY_UID`, default 1337). +3. `RETURN` loopback (the app → proxy hop) and in-cluster CIDRs (`CLUSTER_CIDRS`, mesh/DNS). +4. `DROP` everything else — direct external egress, including UDP (QUIC/HTTP-3). + +An IPv6 mirror drops external v6 egress (allowing loopback, link-local, +the proxy UID, and `CLUSTER_CIDRS6`). + +**Why `mangle` OUTPUT, not `filter`:** when Istio ambient is active it +installs an in-pod `nat OUTPUT` REDIRECT (`ISTIO_OUTPUT` → ztunnel +`:15001`). The netfilter OUTPUT hook order is `raw → mangle → nat → +filter`, so a DROP in `mangle` evaluates the original destination and +fires **before** ambient's nat redirect can rewrite it; a DROP in +`filter` would run after nat and be defeated. `-I 1` also places the +chain ahead of Istio's appended (`-A`) mangle chain. This makes the +guard robust with no ambient, in-pod ambient, or node-level ambient. +See [`test-enforce-drop.sh`](./test-enforce-drop.sh), which proves the +preemption via packet counters. + +## iptables backend + The script auto-detects `iptables-legacy` vs `iptables-nft` and uses -whichever the host kernel exposes. Override with `IPTABLES_CMD` if -needed. +whichever the host kernel exposes. Override with `IPTABLES_CMD` (and +`IP6TABLES_CMD`) if needed. ## Environment variables -| Variable | Default | Purpose | -|---|---|---| -| `PROXY_PORT` | `15123` | AuthBridge outbound listener port | -| `INBOUND_PROXY_PORT` | `15124` | AuthBridge inbound listener port | -| `PROXY_UID` | `1337` | UID of the AuthBridge sidecar process; excluded from outbound redirect | -| `OUTBOUND_PORTS_EXCLUDE` | (empty) | Comma-separated outbound port list to skip (e.g. `8080`) | -| `INBOUND_PORTS_EXCLUDE` | (empty) | Comma-separated inbound port list to skip | -| `POD_IP` | (required) | Set via Downward API; used as the DNAT target for ambient-mesh inbound | -| `IPTABLES_CMD` | auto-detected | Override iptables binary (`iptables-legacy` / `iptables-nft`) | +| Variable | Default | Mode | Purpose | +|---|---|---|---| +| `MODE` | `redirect` | both | `redirect` (envoy-sidecar) or `enforce-drop` (proxy-sidecar) | +| `PROXY_UID` | `1337` | both | UID of the AuthBridge sidecar process; exempted from redirect / drop | +| `PROXY_PORT` | `15123` | redirect | AuthBridge outbound listener port | +| `INBOUND_PROXY_PORT` | `15124` | redirect | AuthBridge inbound listener port | +| `OUTBOUND_PORTS_EXCLUDE` | (empty) | redirect | Comma-separated outbound port list to skip (e.g. `8080`) | +| `INBOUND_PORTS_EXCLUDE` | (empty) | redirect | Comma-separated inbound port list to skip | +| `POD_IP` | (required in `redirect`) | redirect | Set via Downward API; DNAT target for ambient-mesh inbound. Not used by `enforce-drop`. | +| `CLUSTER_CIDRS` | `10.0.0.0/8` | enforce-drop | Comma-separated in-cluster CIDRs allowed direct (pods/services/DNS) | +| `CLUSTER_CIDRS6` | (empty) | enforce-drop | IPv6 in-cluster CIDRs (dual-stack); empty drops all external v6 egress | +| `IPTABLES_CMD` | auto-detected | both | Override iptables binary (`iptables-legacy` / `iptables-nft`) | +| `IP6TABLES_CMD` | derived from `IPTABLES_CMD` | enforce-drop | Override ip6tables binary | ## Required Kubernetes capabilities @@ -62,11 +101,30 @@ The image is published from CI as `ghcr.io/kagenti/kagenti-extensions/proxy-init:` (build defined in [`.github/workflows/build.yaml`](../../.github/workflows/build.yaml)). +## Testing + +[`test-enforce-drop.sh`](./test-enforce-drop.sh) validates `enforce-drop` +mode in a private network namespace (`unshare --net`): it asserts the +`AB_EGRESS` rule structure and proves the `mangle` DROP preempts a +simulated Istio ambient `nat OUTPUT` REDIRECT via packet counters. +Requires root + iptables-nft on Linux (runs on CI; not macOS): + +```sh +sudo ./test-enforce-drop.sh +``` + ## Where it gets injected The kagenti-operator's mutating webhook injects the proxy-init -container automatically when the resolved AuthBridge mode is -`envoy-sidecar`. See +container automatically: + +- `redirect` mode (`MODE` unset) when the resolved AuthBridge mode is + `envoy-sidecar`. +- `enforce-drop` mode (`MODE=enforce-drop`) when `proxy-sidecar` + egress enforcement is enabled (opt-in; see the kagenti-operator + injector wiring). + +See [`authbridge/demos/weather-agent/demo-ui-advanced.md`](../demos/weather-agent/demo-ui-advanced.md) for an end-to-end demo and [`authbridge/demos/token-exchange-routes/README.md`](../demos/token-exchange-routes/README.md) diff --git a/authbridge/proxy-init/init-iptables.sh b/authbridge/proxy-init/init-iptables.sh index 9cd10269e..393206e68 100644 --- a/authbridge/proxy-init/init-iptables.sh +++ b/authbridge/proxy-init/init-iptables.sh @@ -126,6 +126,22 @@ set -e +# --- Mode selection --- +# MODE selects the interception strategy: +# redirect (default) — envoy-sidecar: transparently REDIRECT pod traffic +# to the Envoy listeners (the behavior documented above). +# enforce-drop — proxy-sidecar: a fail-closed egress guard. The app is +# configured with HTTP_PROXY pointing at AuthBridge's forward +# proxy; this mode DROPs any egress that bypasses the proxy, +# forcing all external traffic through AuthBridge regardless of +# whether the app honors HTTP_PROXY. It installs no REDIRECT and +# no PREROUTING/inbound rules. See setup_enforce_drop() below. +MODE="${MODE:-redirect}" +case "${MODE}" in + redirect|enforce-drop) ;; + *) echo "ERROR: unknown MODE='${MODE}' (expected: redirect | enforce-drop)" >&2; exit 1 ;; +esac + # --- Auto-detect iptables backend --- # Prefer iptables-legacy for maximum compatibility with Kubernetes networking. # The nft backend sets rules in a different netfilter table that may not be @@ -153,6 +169,17 @@ SSH_PORT="${SSH_PORT:-22}" OUTBOUND_PORTS_EXCLUDE="${OUTBOUND_PORTS_EXCLUDE:-}" INBOUND_PORTS_EXCLUDE="${INBOUND_PORTS_EXCLUDE:-}" +# enforce-drop mode: in-cluster destinations the agent may reach directly +# (pods / services / DNS) — everything else egressing the pod is dropped. +# Defaults to the RFC1918 10/8 block which covers typical Kind pod (10.244/16) +# and service (10.96/16) CIDRs; override with the cluster's actual ranges. +CLUSTER_CIDRS="${CLUSTER_CIDRS:-10.0.0.0/8}" +CLUSTER_CIDRS6="${CLUSTER_CIDRS6:-}" # IPv6 in-cluster CIDRs (dual-stack); empty = none + +# IPv6 counterpart of the detected iptables backend (iptables-legacy -> +# ip6tables-legacy, iptables -> ip6tables). Override with IP6TABLES_CMD. +IP6T="${IP6TABLES_CMD:-$(echo "${IPT}" | sed 's/iptables/ip6tables/')}" + # Istio ztunnel defaults ZTUNNEL_HBONE_PORT="${ZTUNNEL_HBONE_PORT:-15008}" ZTUNNEL_MARK="${ZTUNNEL_MARK:-0x539/0xfff}" # 0x539 = 1337 decimal, ztunnel's socket fwmark @@ -162,12 +189,105 @@ ISTIO_HEALTH_PROBE_SRC="${ISTIO_HEALTH_PROBE_SRC:-169.254.7.127}" # It must be passed via the Kubernetes Downward API (status.podIP) or set manually. # We use DNAT to the pod IP instead of REDIRECT to avoid needing route_localnet=1, # which would require a privileged init container (to write to read-only /proc/sys). -if [ -z "${POD_IP}" ]; then - echo "ERROR: POD_IP environment variable is not set." >&2 +# POD_IP is only needed by redirect mode (DNAT target for the ambient inbound +# rule). enforce-drop does no DNAT, so it does not require it. +if [ "${MODE}" = "redirect" ] && [ -z "${POD_IP}" ]; then + echo "ERROR: POD_IP environment variable is not set (required for redirect mode)." >&2 echo "Set it via the Kubernetes Downward API (status.podIP) or manually." >&2 exit 1 fi +# ============================================================================= +# enforce-drop mode (proxy-sidecar fail-closed egress guard) +# ============================================================================= +# +# proxy-sidecar configures the app with HTTP_PROXY=127.0.0.1:. +# Unlike redirect mode we do NOT transparently REDIRECT — you cannot redirect +# raw traffic into a CONNECT forward proxy. Instead we DROP any egress that +# leaves the pod without going through the proxy, forcing all external traffic +# through AuthBridge regardless of whether the app honors HTTP_PROXY. +# +# Placement — a dedicated chain hooked from *mangle* OUTPUT at position 1: +# * Istio ambient, when active, installs an in-pod `nat OUTPUT` REDIRECT +# (ISTIO_OUTPUT -> ztunnel :15001). The netfilter OUTPUT hook order is +# raw -> mangle -> nat -> filter, so a DROP in mangle evaluates the +# ORIGINAL destination and fires BEFORE ambient's nat redirect can rewrite +# it. A DROP in `filter` would run after nat and be defeated (dst already +# rewritten to 127.0.0.1). -I 1 also places us ahead of Istio's appended +# (-A) mangle ISTIO_OUTPUT chain. +# * Works identically with no ambient, in-pod ambient, or node-level ambient +# (in the node-level case our pod-netns rule runs before the packet ever +# reaches the host netns). +# +# Rule order in the chain: RETURN ztunnel's own sockets (fwmark 0x539, a no-op +# when ambient is absent) -> RETURN the proxy's own egress (PROXY_UID) -> +# RETURN loopback (app -> proxy) -> RETURN in-cluster CIDRs (mesh/DNS) -> +# DROP everything else (direct external egress, incl. UDP/QUIC). +setup_enforce_drop() { + CHAIN="AB_EGRESS" + + echo "enforce-drop: installing fail-closed egress guard (mangle OUTPUT, chain ${CHAIN})" + echo "enforce-drop: exempt proxy UID=${PROXY_UID}; allowed in-cluster CIDRs=${CLUSTER_CIDRS}" + + # --- IPv4 --- + ${IPT} -t mangle -N "${CHAIN}" 2>/dev/null || true + ${IPT} -t mangle -F "${CHAIN}" + # ztunnel's own sockets (ambient) carry fwmark 0x539 — let them through so the + # mesh/HBONE path keeps working. No-op when ambient is not installed. + ${IPT} -t mangle -A "${CHAIN}" -m mark --mark "${ZTUNNEL_MARK}" -j RETURN + # the AuthBridge proxy's own re-originated egress (must run as PROXY_UID). + ${IPT} -t mangle -A "${CHAIN}" -m owner --uid-owner "${PROXY_UID}" -j RETURN + # app -> proxy over loopback (HTTP_PROXY target), and any loopback traffic. + ${IPT} -t mangle -A "${CHAIN}" -o lo -j RETURN + ${IPT} -t mangle -A "${CHAIN}" -d 127.0.0.0/8 -j RETURN + # in-cluster traffic (pods / services / DNS) — carried by the mesh, not the proxy. + for cidr in $(echo "${CLUSTER_CIDRS}" | tr ',' ' '); do + [ -n "${cidr}" ] && ${IPT} -t mangle -A "${CHAIN}" -d "${cidr}" -j RETURN + done + # everything else == direct external egress that bypassed the proxy. Drop it. + # No -p filter, so UDP (QUIC/HTTP-3) is dropped as well as TCP. + ${IPT} -t mangle -A "${CHAIN}" -j DROP + # Hook at position 1 so we run before any appended Istio mangle chain and + # before nat OUTPUT. + if ! ${IPT} -t mangle -C OUTPUT -j "${CHAIN}" 2>/dev/null; then + ${IPT} -t mangle -I OUTPUT 1 -j "${CHAIN}" + fi + echo "enforce-drop: IPv4 egress guard configured" + + # --- IPv6 --- + # Cluster is IPv4-only by default; until v6 cluster CIDRs are wired + # (CLUSTER_CIDRS6), drop all external v6 egress while allowing loopback, + # link-local (NDP), the proxy UID, and ztunnel's mark. + if command -v "${IP6T%% *}" >/dev/null 2>&1 && ${IP6T} -t mangle -L >/dev/null 2>&1; then + ${IP6T} -t mangle -N "${CHAIN}" 2>/dev/null || true + ${IP6T} -t mangle -F "${CHAIN}" + ${IP6T} -t mangle -A "${CHAIN}" -m mark --mark "${ZTUNNEL_MARK}" -j RETURN + ${IP6T} -t mangle -A "${CHAIN}" -m owner --uid-owner "${PROXY_UID}" -j RETURN + ${IP6T} -t mangle -A "${CHAIN}" -o lo -j RETURN + ${IP6T} -t mangle -A "${CHAIN}" -d ::1/128 -j RETURN + ${IP6T} -t mangle -A "${CHAIN}" -d fe80::/10 -j RETURN + for cidr in $(echo "${CLUSTER_CIDRS6}" | tr ',' ' '); do + [ -n "${cidr}" ] && ${IP6T} -t mangle -A "${CHAIN}" -d "${cidr}" -j RETURN + done + ${IP6T} -t mangle -A "${CHAIN}" -j DROP + if ! ${IP6T} -t mangle -C OUTPUT -j "${CHAIN}" 2>/dev/null; then + ${IP6T} -t mangle -I OUTPUT 1 -j "${CHAIN}" + fi + echo "enforce-drop: IPv6 egress guard configured" + else + echo "enforce-drop: ip6tables unavailable — skipping IPv6 egress guard" + fi + + echo "enforce-drop: fail-closed egress guard active" +} + +# Dispatch enforce-drop here and exit; redirect mode falls through to the +# transparent-interception logic below. +if [ "${MODE}" = "enforce-drop" ]; then + setup_enforce_drop + exit 0 +fi + # ============================================================================= # OUTBOUND traffic interception (nat OUTPUT) # ============================================================================= diff --git a/authbridge/proxy-init/test-enforce-drop.sh b/authbridge/proxy-init/test-enforce-drop.sh new file mode 100755 index 000000000..acc71518d --- /dev/null +++ b/authbridge/proxy-init/test-enforce-drop.sh @@ -0,0 +1,87 @@ +#!/usr/bin/env bash +# +# Test harness for init-iptables.sh "enforce-drop" mode (proxy-sidecar +# fail-closed egress guard). +# +# It validates two things in a private network namespace: +# 1. Rule STRUCTURE — the AB_EGRESS chain is hooked from mangle OUTPUT at +# position 1 with the expected RETURN exemptions and a terminal DROP, and +# that no nat/filter rules are created. +# 2. AMBIENT ROBUSTNESS — a DROP in mangle OUTPUT preempts a simulated Istio +# ambient "nat OUTPUT REDIRECT" (ISTIO_OUTPUT). Proven via packet counters: +# after generating an external SYN, the mangle DROP increments and the nat +# REDIRECT does NOT. +# +# Requirements: root (for unshare --net + iptables), iproute2, iptables-nft, +# bash, the dummy kernel module. Runs on Linux / CI (e.g. ubuntu-latest); not +# on macOS. Uses `unshare --net` (not named `ip netns`) so it also works inside +# nested containers. Exit code 0 = all pass. +set -u + +SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +INIT="${INIT_SCRIPT:-${SCRIPT_DIR}/init-iptables.sh}" +IPT="${IPTABLES_CMD:-iptables-nft}" +EXTERNAL="198.51.100.7" # RFC5737 TEST-NET-2, guaranteed unused + +# Re-exec into a private network namespace. unshare avoids the /sys remount that +# named `ip netns exec` performs, so this works inside nested containers too. +if [ -z "${_AB_NETNS_REEXEC:-}" ]; then + exec unshare --net env _AB_NETNS_REEXEC=1 INIT_SCRIPT="${INIT}" \ + IPTABLES_CMD="${IPT}" bash "$0" "$@" +fi + +fail=0 + +# Fresh netns: bring up lo and a dummy default route so packets to an external +# destination are actually generated and traverse the OUTPUT chain. +ip link set lo up +if ip link add eth-test type dummy 2>/dev/null; then + ip addr add 10.255.255.2/24 dev eth-test + ip link set eth-test up + ip route add default via 10.255.255.1 +else + echo "WARN: dummy interface unavailable; preemption packet may not be generated" +fi + +echo "### Installing enforce-drop rules" +env MODE=enforce-drop PROXY_UID=1337 CLUSTER_CIDRS=10.0.0.0/8 \ + IPTABLES_CMD="${IPT}" IP6TABLES_CMD=ip6tables-nft \ + sh "${INIT}" || { echo "FAIL: init script exited non-zero"; exit 1; } + +dump=$("${IPT}" -t mangle -S) +echo "--- mangle ruleset ---"; echo "${dump}" + +assert() { if echo "${dump}" | grep -qE "$2"; then echo "PASS: $1"; else echo "FAIL: $1"; fail=1; fi; } +assert "AB_EGRESS hooked from OUTPUT" '^-A OUTPUT -j AB_EGRESS' +assert "ztunnel mark RETURN" 'AB_EGRESS .*mark.*0x539.*-j RETURN' +assert "proxy UID RETURN" 'AB_EGRESS .*--uid-owner 1337 -j RETURN' +assert "loopback iface RETURN" 'AB_EGRESS -o lo -j RETURN' +assert "loopback cidr RETURN" 'AB_EGRESS -d 127.0.0.0/8 -j RETURN' +assert "cluster cidr RETURN" 'AB_EGRESS -d 10.0.0.0/8 -j RETURN' +assert "terminal DROP" 'AB_EGRESS -j DROP' + +pos1=$("${IPT}" -t mangle -L OUTPUT --line-numbers -n | awk '$1=="1"{print $2}') +if [ "${pos1}" = "AB_EGRESS" ]; then echo "PASS: AB_EGRESS at OUTPUT position 1" +else echo "FAIL: AB_EGRESS not at OUTPUT position 1 (got '${pos1}')"; fail=1; fi + +natcount=$("${IPT}" -t nat -S | grep -cE 'AB_EGRESS|REDIRECT|PROXY_' || true) +if [ "${natcount:-0}" -eq 0 ]; then echo "PASS: no nat-table rules created" +else echo "FAIL: enforce-drop created nat rules"; fail=1; fi + +echo "### Ambient-preemption test: append a simulated ISTIO_OUTPUT nat REDIRECT" +"${IPT}" -t nat -A OUTPUT -p tcp -d "${EXTERNAL}" -j REDIRECT --to-ports 19999 +# Generate an external SYN (uid 0, like an agent bypass attempt). +timeout 2 bash -c "exec 3<>/dev/tcp/${EXTERNAL}/80" 2>/dev/null || true + +dropc=$("${IPT}" -t mangle -L AB_EGRESS -n -v | awk '/DROP/{print $1; exit}') +redirc=$("${IPT}" -t nat -L OUTPUT -n -v | awk '/REDIRECT/{print $1; exit}') +echo "mangle AB_EGRESS DROP pkts=${dropc:-?} | nat REDIRECT pkts=${redirc:-?}" +if [ "${dropc:-0}" -gt 0 ] && [ "${redirc:-0}" -eq 0 ]; then + echo "PASS: mangle DROP preempted nat REDIRECT (ambient-robust)" +else + echo "FAIL: preemption not demonstrated (DROP=${dropc:-?}, REDIRECT=${redirc:-?})"; fail=1 +fi + +echo +[ "${fail}" -eq 0 ] && echo "ALL TESTS PASSED" || echo "SOME TESTS FAILED" +exit "${fail}" From 17c05ec1ae87bee68d572240f6db7653f04301ec Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Fri, 5 Jun 2026 13:35:40 -0400 Subject: [PATCH 2/4] fix(proxy-init): let established/related replies through enforce-drop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address review on #484: the AB_EGRESS chain filtered purely on dest/owner/mark, so reply packets for inbound connections (which also traverse mangle OUTPUT) to an off-cluster peer would hit the terminal DROP — e.g. kubelet health-probe responses to the Kind node IP (172.18.0.0/16, outside the default CLUSTER_CIDRS=10.0.0.0/8), causing probe failures and pod restarts once enforcement is enabled. - Add `-m conntrack --ctstate ESTABLISHED,RELATED -j RETURN` as the first rule (IPv4 + IPv6). Only NEW app-initiated flows are gated; a reply is never a bypass. - IPv6: also allow link-local multicast (ff02::/16) so NDP neighbor/router solicitations work, and correct the comment that overstated NDP preservation. - README: clarify the enforce-drop injection lands in the follow-up operator PR rather than reading as already-available. - test: assert the established/related RETURN exists and is the first rule in AB_EGRESS. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- authbridge/proxy-init/README.md | 5 +++-- authbridge/proxy-init/init-iptables.sh | 14 ++++++++++++-- authbridge/proxy-init/test-enforce-drop.sh | 7 +++++++ 3 files changed, 22 insertions(+), 4 deletions(-) diff --git a/authbridge/proxy-init/README.md b/authbridge/proxy-init/README.md index 84025390c..09b9b6f87 100644 --- a/authbridge/proxy-init/README.md +++ b/authbridge/proxy-init/README.md @@ -121,8 +121,9 @@ container automatically: - `redirect` mode (`MODE` unset) when the resolved AuthBridge mode is `envoy-sidecar`. - `enforce-drop` mode (`MODE=enforce-drop`) when `proxy-sidecar` - egress enforcement is enabled (opt-in; see the kagenti-operator - injector wiring). + egress enforcement is enabled (opt-in). _The operator wiring that + sets this lands in the follow-up kagenti-operator PR; this PR only + adds the mode to the image._ See [`authbridge/demos/weather-agent/demo-ui-advanced.md`](../demos/weather-agent/demo-ui-advanced.md) diff --git a/authbridge/proxy-init/init-iptables.sh b/authbridge/proxy-init/init-iptables.sh index 393206e68..71fa67835 100644 --- a/authbridge/proxy-init/init-iptables.sh +++ b/authbridge/proxy-init/init-iptables.sh @@ -232,6 +232,12 @@ setup_enforce_drop() { # --- IPv4 --- ${IPT} -t mangle -N "${CHAIN}" 2>/dev/null || true ${IPT} -t mangle -F "${CHAIN}" + # Replies to inbound connections (and related flows) are locally generated and + # also traverse OUTPUT — a reply is never a "bypass". Let established/related + # traffic through FIRST, so e.g. kubelet health-probe responses to an + # off-cluster node IP (in Kind the node is 172.18.0.0/16, outside CLUSTER_CIDRS) + # are not caught by the terminal DROP. Only NEW app-initiated flows are gated. + ${IPT} -t mangle -A "${CHAIN}" -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN # ztunnel's own sockets (ambient) carry fwmark 0x539 — let them through so the # mesh/HBONE path keeps working. No-op when ambient is not installed. ${IPT} -t mangle -A "${CHAIN}" -m mark --mark "${ZTUNNEL_MARK}" -j RETURN @@ -256,16 +262,20 @@ setup_enforce_drop() { # --- IPv6 --- # Cluster is IPv4-only by default; until v6 cluster CIDRs are wired - # (CLUSTER_CIDRS6), drop all external v6 egress while allowing loopback, - # link-local (NDP), the proxy UID, and ztunnel's mark. + # (CLUSTER_CIDRS6), drop external v6 egress while allowing: established/related + # replies, loopback, link-local unicast (fe80::/10) and link-local multicast + # (ff02::/16, which carries NDP neighbor/router solicitations and MLD), the + # proxy UID, and ztunnel's mark. if command -v "${IP6T%% *}" >/dev/null 2>&1 && ${IP6T} -t mangle -L >/dev/null 2>&1; then ${IP6T} -t mangle -N "${CHAIN}" 2>/dev/null || true ${IP6T} -t mangle -F "${CHAIN}" + ${IP6T} -t mangle -A "${CHAIN}" -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN ${IP6T} -t mangle -A "${CHAIN}" -m mark --mark "${ZTUNNEL_MARK}" -j RETURN ${IP6T} -t mangle -A "${CHAIN}" -m owner --uid-owner "${PROXY_UID}" -j RETURN ${IP6T} -t mangle -A "${CHAIN}" -o lo -j RETURN ${IP6T} -t mangle -A "${CHAIN}" -d ::1/128 -j RETURN ${IP6T} -t mangle -A "${CHAIN}" -d fe80::/10 -j RETURN + ${IP6T} -t mangle -A "${CHAIN}" -d ff02::/16 -j RETURN for cidr in $(echo "${CLUSTER_CIDRS6}" | tr ',' ' '); do [ -n "${cidr}" ] && ${IP6T} -t mangle -A "${CHAIN}" -d "${cidr}" -j RETURN done diff --git a/authbridge/proxy-init/test-enforce-drop.sh b/authbridge/proxy-init/test-enforce-drop.sh index acc71518d..606f42fe1 100755 --- a/authbridge/proxy-init/test-enforce-drop.sh +++ b/authbridge/proxy-init/test-enforce-drop.sh @@ -53,6 +53,7 @@ echo "--- mangle ruleset ---"; echo "${dump}" assert() { if echo "${dump}" | grep -qE "$2"; then echo "PASS: $1"; else echo "FAIL: $1"; fail=1; fi; } assert "AB_EGRESS hooked from OUTPUT" '^-A OUTPUT -j AB_EGRESS' +assert "established/related RETURN" 'AB_EGRESS -m conntrack --ctstate (ESTABLISHED,RELATED|RELATED,ESTABLISHED) -j RETURN' assert "ztunnel mark RETURN" 'AB_EGRESS .*mark.*0x539.*-j RETURN' assert "proxy UID RETURN" 'AB_EGRESS .*--uid-owner 1337 -j RETURN' assert "loopback iface RETURN" 'AB_EGRESS -o lo -j RETURN' @@ -64,6 +65,12 @@ pos1=$("${IPT}" -t mangle -L OUTPUT --line-numbers -n | awk '$1=="1"{print $2}') if [ "${pos1}" = "AB_EGRESS" ]; then echo "PASS: AB_EGRESS at OUTPUT position 1" else echo "FAIL: AB_EGRESS not at OUTPUT position 1 (got '${pos1}')"; fail=1; fi +# the established/related RETURN must be the first rule in the chain (replies +# must be let through before any owner/dest evaluation). +first_rule=$("${IPT}" -t mangle -S AB_EGRESS | grep '^-A AB_EGRESS' | head -1) +if echo "${first_rule}" | grep -q 'conntrack'; then echo "PASS: established/related RETURN is first in AB_EGRESS" +else echo "FAIL: first AB_EGRESS rule is not the conntrack RETURN (got: ${first_rule})"; fail=1; fi + natcount=$("${IPT}" -t nat -S | grep -cE 'AB_EGRESS|REDIRECT|PROXY_' || true) if [ "${natcount:-0}" -eq 0 ]; then echo "PASS: no nat-table rules created" else echo "FAIL: enforce-drop created nat rules"; fail=1; fi From 4f66a6b3a2ab27f2fb08d575a0cee3afb5119a5d Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Fri, 5 Jun 2026 13:44:45 -0400 Subject: [PATCH 3/4] test(proxy-init): enable strict mode in enforce-drop harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address CodeRabbit on #484: switch the test harness from set -u to set -euo pipefail (the repo's shell convention) so command/pipe failures in the setup phase surface instead of being masked. The run-all-assertions design is unaffected — asserts return 0 via if/else and expected-nonzero commands are guarded with || true; verified all 13 assertions still run and the harness exits 0. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- authbridge/proxy-init/test-enforce-drop.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/authbridge/proxy-init/test-enforce-drop.sh b/authbridge/proxy-init/test-enforce-drop.sh index 606f42fe1..0cf570fd2 100755 --- a/authbridge/proxy-init/test-enforce-drop.sh +++ b/authbridge/proxy-init/test-enforce-drop.sh @@ -16,7 +16,7 @@ # bash, the dummy kernel module. Runs on Linux / CI (e.g. ubuntu-latest); not # on macOS. Uses `unshare --net` (not named `ip netns`) so it also works inside # nested containers. Exit code 0 = all pass. -set -u +set -euo pipefail SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) INIT="${INIT_SCRIPT:-${SCRIPT_DIR}/init-iptables.sh}" From f3101d7bf8d036679cf389b463d013d78d0b5a74 Mon Sep 17 00:00:00 2001 From: Hai Huang Date: Fri, 5 Jun 2026 14:03:28 -0400 Subject: [PATCH 4/4] docs(proxy-init): clarify CLUSTER_CIDRS portability and enforce-drop semantics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address mrsabath review on #484 (non-blocking): - CLUSTER_CIDRS=10.0.0.0/8 default is Kind-shaped; OCP (services 172.30.0.0/16, pods 10.128.0.0/14) and EKS users must override it or in-cluster service traffic is dropped. Operator wiring (follow-up PR) sets it from the cluster's real CIDRs. - enforce-drop intentionally ignores OUTBOUND_PORTS_EXCLUDE (redirect-only); a destination previously bypassed that way is now dropped unless routed through the proxy or in CLUSTER_CIDRS — by design. The third suggestion (set -euo pipefail) already landed in 4f66a6b. Assisted-By: Claude (Anthropic AI) Signed-off-by: Hai Huang --- authbridge/proxy-init/README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/authbridge/proxy-init/README.md b/authbridge/proxy-init/README.md index 09b9b6f87..0bb03d7c9 100644 --- a/authbridge/proxy-init/README.md +++ b/authbridge/proxy-init/README.md @@ -50,6 +50,23 @@ proxy regardless of whether the app honors `HTTP_PROXY`. An IPv6 mirror drops external v6 egress (allowing loopback, link-local, the proxy UID, and `CLUSTER_CIDRS6`). +> **`CLUSTER_CIDRS` is Kind-shaped by default.** The `10.0.0.0/8` default +> covers Kind (pods `10.244.0.0/16` + services `10.96.0.0/16`). Other +> distros differ — **OpenShift** uses services `172.30.0.0/16` and pods +> `10.128.0.0/14`, and `172.30.0.0/16` is **outside** `10/8`, so the +> default would drop in-cluster service traffic. On OCP/EKS/etc. you +> **must** override `CLUSTER_CIDRS` with the cluster's real pod+service +> ranges. The script logs the resolved value at startup, and the +> operator wiring (follow-up PR) sets it from the cluster's CIDRs. + +> **`enforce-drop` intentionally ignores `OUTBOUND_PORTS_EXCLUDE`** (a +> `redirect`-mode knob). Any destination previously bypassed that way — +> e.g. a direct LLM endpoint at `host.docker.internal:11434` — is now +> dropped unless it goes through the forward proxy or falls within +> `CLUSTER_CIDRS`. That is the point: `enforce-drop` closes direct-egress +> holes. Operators relying on a bypass must route it through the proxy +> (or, for in-cluster targets, include it in `CLUSTER_CIDRS`). + **Why `mangle` OUTPUT, not `filter`:** when Istio ambient is active it installs an in-pod `nat OUTPUT` REDIRECT (`ISTIO_OUTPUT` → ztunnel `:15001`). The netfilter OUTPUT hook order is `raw → mangle → nat →