Pin Ziti enrollment releases and configure workload egress

[rowan-stein]

User request

E2E tests that start workloads and wait for completion must fully work. agynio/e2e PR #214 now exposes the live workload/Ziti failure, but the remaining fix must be made in upstream platform bootstrap/pins/config rather than E2E-only workflow patches.

Related tracking:

• E2E issue: Fix full E2E coverage for AGN CLI, platform egress, and real-agent Ziti sidecars e2e#213
• E2E PR: fix(e2e): restore real agent CLI coverage e2e#214
• Latest failing run: https://github.com/agynio/e2e/actions/runs/28344843464
• agents-orchestrator tracker: Fix workload Ziti sidecar initialization contract agents-orchestrator#219
• k8s-runner tracker: Add workload egress allowances for Ziti enrollment k8s-runner#73

Specification

Architecture-backed diagnosis:

• Agents Orchestrator owns OpenZiti identity creation and enrollment JWT handoff.
• k8s-runner chart owns workload egress NetworkPolicy templates.
• bootstrap owns platform source-of-truth pins/config.
• agynio/e2e#214 should remain coverage/diagnostics only.

Observed failure:

failed to parse JWT: token is unverifiable: error while executing keyfunc: could not retrieve token URL certificate: could not contact remote server [https://ziti.agyn.dev:2496]: Get "https://ziti.agyn.dev:2496": dial tcp 10.43.245.186:2496: connect: connection refused

Required bootstrap changes:

• Repo: agynio/bootstrap
• Files likely involved:
  • stacks/platform/variables.tf
  • stacks/platform/main.tf
  • stacks/platform/ziti_workload_dns.tf
  • stacks/apps/variables.tf
  • apps/platform Helm values wiring for k8s-runner / agents-orchestrator

Scope:

1. Bump agents_orchestrator_chart_version from current 0.13.18 to a release containing the explicit ziti-enroll init-container fix from agents-orchestrator main, including commit c539ac8 and current DNS/enrollment behavior.
2. Bump k8s_runner_chart_version if required after agynio/k8s-runner#73 lands.
3. Preserve/configure WORKLOAD_DNS_UPSTREAM for agents-orchestrator using the ziti-workload-dns service IP/source-of-truth value.
4. Configure k8s-runner workload egress policy values for:
   • Ziti workload DNS TCP/UDP 53.
   • Ziti controller enrollment TCP 2496 or configured ingress port to ziti-controller-client ClusterIP /32.

Acceptance criteria:

• Terraform plan/render shows agents-orchestrator pinned to a release containing explicit ziti-enroll.
• k8s-runner chart values include Ziti workload DNS and controller enrollment egress allowances.
• ziti-workload-dns continues to map ziti.agyn.dev to the controller service as intended.
• After apply/sync, workload pods can complete ziti-enroll, start sidecar, pass gateway wait, and run agents.
• agynio/e2e PR fix: update threads chart tag #214 full matrix passes:
  • platform-usage
  • k8s-runner-workloads
  • orchestrator-workloads
  • real-agent-ziti

Suggested verification:

terraform -chdir=stacks/platform plan
terraform -chdir=stacks/apps plan

Check planned values include:

• agents_orchestrator_chart_version = <new fixed release>
• k8s-runner version/value changes if applicable
• WORKLOAD_DNS_UPSTREAM=<ziti-workload-dns ClusterIP>
• workload egress policy values for Ziti DNS and controller enrollment

After deploy/sync:

kubectl get configmap ziti-workload-dns -n ziti -o yaml
kubectl get networkpolicy agent-workload-egress -n agyn-workloads -o yaml
kubectl get deploy agents-orchestrator -n platform -o jsonpath='{.spec.template.spec.containers[0].image}'
kubectl get app agents-orchestrator -n argocd
kubectl get app k8s-runner -n argocd

Then rerun targeted agynio/e2e workload groups and the full PR #214 matrix.

[rowan-stein]

Architecture-backed follow-up from the latest PR220 / PR214 live E2E failures.

Current status:

• agents-orchestrator PR Add dedicated OIDC client ID for console-app #220 now gets past enrollment.
• ziti-enroll completes successfully.
• Runtime ziti-sidecar still cannot reach the Ziti controller/router underlay after loading identity.

Latest symptom:

ziti-sidecar log: loading identity: /netfoundry/agent.json
ziti-sidecar log: Get "https://ziti.agyn.dev:2496/edge/client/v1/version": dial tcp <ziti-controller-client ClusterIP>:2496: connect: connection refused
ziti-gateway-wait log: timeout waiting for gateway.ziti

Bootstrap should provide the source-of-truth config for the complete Ziti workload networking contract:

1. Align OpenZiti releases:

   • controller chart default to released 3.2.0
   • router chart default to released 3.0.0
   • remove or update stale ZITI_SIDECAR_IMAGE override to openziti/ziti-tunnel:2.0.0

2. Add ZITI_ENROLLMENT_DNS_UPSTREAM for agents-orchestrator values:

   • should point to the normal cluster/CoreDNS resolver used before the tunnel is running
   • keep WORKLOAD_DNS_UPSTREAM pointing to ziti-workload-dns for runtime sidecar .ziti DNS

3. Wire k8s-runner workload egress values once PR Upgrade gateway to v0.3.0 #74 supports the complete underlay path:

   • DNS to ziti-workload-dns
   • controller underlay endpoint/port returned for ziti.agyn.dev
   • router underlay endpoint/port returned for ziti-router.agyn.dev

Do not broadly allow private/service CIDRs. Prefer explicit service /32 + port values or selector-based values, depending on what k8s-runner chart can support reliably in the cluster CNI.

Validation target:

nslookup ziti.agyn.dev <ziti-workload-dns-ip>
nc -vz ziti.agyn.dev 2496
nslookup ziti-router.agyn.dev <ziti-workload-dns-ip>
nc -vz ziti-router.agyn.dev 2496

This issue should remain the bootstrap source-of-truth follow-up after k8s-runner PR #74 and agents-orchestrator PR #220 are corrected/released.

[rowan-stein]

Architecture-first update from the latest agents-orchestrator#220 E2E iteration:

The old acceptance detail that ziti-workload-dns should map ziti.agyn.dev to the controller service is no longer correct for the runtime sidecar path after the identity/runtime API is set to https://ziti.agyn.dev:443/edge/client/v1.

Current proven state:

• ziti-enroll exit=0; enrollment is working.
• Runtime ziti-sidecar fails with:

Get "https://ziti.agyn.dev:443/edge/client/v1/version": dial tcp <cluster-ip>:443: connect: connection refused

Correct split:

1. Enrollment path before tunnel exists:
   • connect to ziti-controller-client.ziti.svc.cluster.local:2496 / local.ingress_port
   • preserve advertised/SNI host ziti.<base_domain>
2. Runtime sidecar path:
   • identity ztAPI=https://ziti.<base_domain>:443/edge/client/v1
   • workload pod DNS for ziti.<base_domain> should resolve to istio-ingressgateway.<istio_gateway_namespace>.svc.cluster.local ClusterIP
   • Istio TLS passthrough on SNI ziti.<base_domain> routes to ziti-controller-client:2496

Required bootstrap change:

• In stacks/platform/ziti_workload_dns.tf, map the runtime ziti.<base_domain> record to the istio-ingressgateway ClusterIP, not ziti-controller-client ClusterIP.
• In stacks/platform/main.tf, pass explicit orchestrator env:
  • ZITI_ENROLLMENT_CONTROLLER_RESOLVE_HOST=ziti-controller-client.<ziti_namespace>.svc.cluster.local
  • ZITI_ENROLLMENT_CONTROLLER_PORT=tostring(local.ingress_port)
  • ZITI_RUNTIME_CONTROLLER_RESOLVE_HOST=istio-ingressgateway.<istio_gateway_namespace>.svc.cluster.local
  • ZITI_RUNTIME_CONTROLLER_PORT=443
• In k8s-runner/bootstrap egress values, allow workload egress to ziti-workload-dns TCP/UDP 53, ziti-controller-client TCP 2496, and istio-ingressgateway TCP 443.

Validation should prove:

ziti-enroll exit=0
ziti-sidecar stays running
gateway.ziti resolves
ziti-gateway-wait exit=0
agent container running
message sent -> agent answer received