feat: wire terminal proxy bootstrap

[casey-brooks]

Summary

Implements the bootstrap deployment wiring slice for agynio/architecture#162.

• Adds the terminal-proxy OpenZiti identity with role attribute terminal-proxy-hosts and exports its enrollment token.
• Deploys Terminal Proxy as a dedicated Argo CD application using the agyn-platform chart Terminal Proxy slice.
• Creates the stable terminal-proxy-ticket-signing Secret and terminal-proxy-ziti-identity Secret for the enrolled identity JSON file.
• Wires Terminal Proxy config/env for Gateway, Runners, Agents, Authorization, WebSocket URL, signing key, and Ziti identity file.
• Wires Gateway with TERMINAL_PROXY_GRPC_TARGET=terminal-proxy:50051.
• Adds the wss://terminal.<base_domain>/terminal Istio route to terminal-proxy.platform.svc.cluster.local:8080.

Validation

• terraform -chdir=stacks/ziti fmt — passed
• terraform -chdir=stacks/platform fmt — passed
• terraform -chdir=stacks/routing fmt — passed
• terraform -chdir=stacks/ziti init -backend=false — passed
• terraform -chdir=stacks/ziti validate — passed
• terraform -chdir=stacks/platform init -backend=false — passed
• terraform -chdir=stacks/platform validate — passed
• terraform -chdir=stacks/routing init -backend=false — passed
• terraform -chdir=stacks/routing validate — passed
• git diff --check — passed

terraform plan -var='terminal_proxy_ziti_identity_json={}' -refresh=false -out=/tmp/platform.tfplan was attempted and could not complete in this workspace because local remote state and kubeconfig are not present (stacks/k8s, stacks/system, and stacks/ziti state are missing; ../k8s/.kube/agyn-local-kubeconfig.yaml does not exist). The partial plan confirmed the new Terminal Proxy signing secret and Ziti identity secret resources before provider/state lookup failed.

Dependency order

Depends on agynio/platform-charts#30 landing first and publishing agyn-platform chart 0.5.11; this PR references that chart version for the Terminal Proxy application.

[casey-brooks]

Validation summary

• terraform -chdir=stacks/ziti fmt — passed
• terraform -chdir=stacks/platform fmt — passed
• terraform -chdir=stacks/routing fmt — passed
• terraform -chdir=stacks/ziti init -backend=false — passed
• terraform -chdir=stacks/ziti validate — passed
• terraform -chdir=stacks/platform init -backend=false — passed
• terraform -chdir=stacks/platform validate — passed
• terraform -chdir=stacks/routing init -backend=false — passed
• terraform -chdir=stacks/routing validate — passed
• git diff --check — passed

terraform plan -var='terminal_proxy_ziti_identity_json={}' -refresh=false -out=/tmp/platform.tfplan was attempted and could not complete in this workspace because local remote state and kubeconfig are not present (stacks/k8s, stacks/system, and stacks/ziti state are missing; ../k8s/.kube/agyn-local-kubeconfig.yaml does not exist). The partial plan confirmed the new Terminal Proxy signing secret and Ziti identity secret resources before provider/state lookup failed.

Dependency note: this PR depends on agynio/platform-charts#30 landing first and publishing agyn-platform chart 0.5.11.

[casey-brooks]

Fix update

Addressed the CI failure where terminal_proxy_ziti_identity_json was unset during ./apply.sh -y.

What changed:

• apply.sh now preserves the pre-provisioned identity-file design while deriving the required Terraform variable during normal bootstrap flow.
• If TERMINAL_PROXY_ZITI_IDENTITY_JSON is supplied, apply.sh exports it as TF_VAR_terminal_proxy_ziti_identity_json.
• If it is not supplied, after the ziti stack succeeds apply.sh enrolls the pre-provisioned terminal-proxy Ziti identity from terminal_proxy_enrollment_token and exports the resulting identity JSON for the platform stack.
• If a terminal-proxy-ziti-identity Secret already exists, apply.sh reuses it instead of reenrolling.
• Documented the derived/env-supplied identity behavior in stacks/platform/README.md.

Validation summary

• terraform -chdir=stacks/ziti init -backend=false — passed
• terraform -chdir=stacks/ziti validate — passed
• terraform -chdir=stacks/platform init -backend=false — passed
• terraform -chdir=stacks/platform validate — passed
• terraform -chdir=stacks/routing init -backend=false — passed
• terraform -chdir=stacks/routing validate — passed
• bash -n apply.sh — passed
• git diff --check — passed

Lint/status:

• Shell syntax lint passed with no errors.
• Terraform validation passed for 3 stacks / 0 failed / 0 skipped.

Dependency note: agynio/platform-charts#30 remains the required dependency before this PR can be applied successfully because bootstrap still references the new agyn-platform chart version 0.5.11 from that PR.

[casey-brooks]

Updated this PR to remove the rejected enrolled identity JSON roundtrip.

Pattern used:

• Removed the terminal_proxy_ziti_identity_json platform variable entirely.
• Removed TERMINAL_PROXY_ZITI_IDENTITY_JSON, capture_existing_terminal_proxy_ziti_identity, and the apply.sh enrollment/identity-json handoff step.
• The ziti stack remains the owner of the terminal-proxy Ziti identity and exposes terminal_proxy_enrollment_token from remote state.
• The platform stack now creates terminal-proxy-enrollment with key enrollmentJwt from that remote-state token, matching the existing ziti-management-enrollment and egress-gateway-enrollment Kubernetes Secret pattern.
• Terminal Proxy chart values mount that enrollment Secret into an init container, which writes /var/run/agyn/terminal-proxy-ziti/identity.json on an emptyDir; the runtime container mounts that identity directory read-only and uses ZITI_IDENTITY_FILE.
• No Orchestrator identity is used.

This keeps Terraform/apply.sh out of enrolled identity JSON capture and lets deployment/runtime wiring produce the identity file from the ziti-owned enrollment token.

Validation summary:

• bash -n apply.sh: passed
• terraform -chdir=stacks/platform fmt: passed
• terraform -chdir=stacks/ziti init -backend=false: passed
• terraform -chdir=stacks/ziti validate: 1 passed, 0 failed
• terraform -chdir=stacks/platform init -backend=false: passed
• terraform -chdir=stacks/platform validate: 1 passed, 0 failed
• terraform -chdir=stacks/routing init -backend=false: passed
• terraform -chdir=stacks/routing validate: 1 passed, 0 failed
• git diff --check: passed

Dependency note: platform-charts#30 still must land/publish first because this stack references the agyn-platform chart version from that PR.

[casey-brooks]

Applied the requested docs cleanup.

• Removed the entire ## Terminal Proxy deployment contract section from stacks/platform/README.md.
• Did not replace it with another contract block.
• Kept the implementation corrections intact: no terminal_proxy_ziti_identity_json variable, no TERMINAL_PROXY_ZITI_IDENTITY_JSON, no apply.sh identity JSON capture/enrollment roundtrip, and platform stack still uses the ziti remote-state enrollment token to create terminal-proxy-enrollment/enrollmentJwt.

Validation summary:

• bash -n apply.sh: passed
• terraform -chdir=stacks/platform fmt: passed
• terraform -chdir=stacks/platform validate: 1 passed, 0 failed
• git diff --check: passed

Pattern remains unchanged: ziti stack owns the terminal-proxy Ziti identity/enrollment token, platform stack creates a Kubernetes enrollment Secret from ziti remote state using the existing bootstrap enrollment-secret convention, and runtime chart wiring enrolls the identity file from that JWT.

[casey-brooks]

Synced the Terminal Proxy identity readability fix into bootstrap chart values.

Change:

• The Terminal Proxy enrollment init container in stacks/platform/main.tf now runs chown 10001:10001 /var/run/agyn/terminal-proxy-ziti/identity.json before chmod 0400, matching platform-charts#30 so UID 10001 can read the enrolled identity file at runtime.

Cleanup remains intact: no terminal_proxy_ziti_identity_json, no TERMINAL_PROXY_ZITI_IDENTITY_JSON, no apply.sh identity JSON roundtrip, and no README deployment contract block.

Validation summary:

• bash -n apply.sh: passed
• terraform -chdir=stacks/platform fmt: passed
• terraform -chdir=stacks/platform validate: 1 passed, 0 failed
• git diff --check: passed

[casey-brooks]

Update

Fixed the Terminal Proxy init-container CrashLoopBackOff root cause.

Root cause

The openziti/ziti-cli:2.0.0-pre10 image defaults to non-root UID/GID 2171:2171. The rendered init command then ran:

chown 10001:10001 /var/run/agyn/terminal-proxy-ziti/identity.json

Because the init container was non-root, chown failed with Operation not permitted, exiting the enroll-ziti-identity init container with code 1 after enrollment wrote the file.

Changes

• stacks/platform/main.tf: set the enroll-ziti-identity init container securityContext to runAsUser: 0 and runAsGroup: 0, preserving the chart contract that enrollment creates identity.json, then chowns it to Terminal Proxy runtime UID 10001 and sets 0400 permissions.
• .github/scripts/verify_platform_health.sh: added focused Terminal Proxy current/previous logs to failure diagnostics so future init-container stderr is visible.

This keeps the existing bootstrap convention: Ziti stack owns the terminal-proxy identity and enrollment token, platform mounts terminal-proxy-enrollment, and runtime init produces identity.json on an emptyDir. It does not reintroduce terminal_proxy_ziti_identity_json, TERMINAL_PROXY_ZITI_IDENTITY_JSON, or capture_existing_terminal_proxy_ziti_identity.

Test & Lint Summary

Commands run:

terraform fmt -recursive -check .
bash -n apply.sh .github/scripts/verify_platform_health.sh
shellcheck apply.sh .github/scripts/verify_platform_health.sh
for stack in deps k8s system routing data ziti platform apps; do terraform -chdir="stacks/${stack}" init -backend=false -input=false && terraform -chdir="stacks/${stack}" validate; done
docker run --rm --user 2171:2171 --entrypoint sh openziti/ziti-cli:2.0.0-pre10 -ec 'mkdir -p /tmp/terminal-proxy-ziti; touch /tmp/terminal-proxy-ziti/identity.json; chown 10001:10001 /tmp/terminal-proxy-ziti/identity.json'
docker run --rm --user 0:0 --entrypoint sh openziti/ziti-cli:2.0.0-pre10 -ec 'mkdir -p /tmp/terminal-proxy-ziti; touch /tmp/terminal-proxy-ziti/identity.json; chown 10001:10001 /tmp/terminal-proxy-ziti/identity.json; chmod 0400 /tmp/terminal-proxy-ziti/identity.json; test "$(stat -c %u:%g /tmp/terminal-proxy-ziti/identity.json)" = "10001:10001"; test "$(stat -c %a /tmp/terminal-proxy-ziti/identity.json)" = "400"'
git diff --check

Results:

• Terraform validation: 8 passed / 0 failed / 0 skipped.
• Shell syntax/lint: 2 scripts passed / 0 failed / 0 skipped.
• Docker reproduction: non-root chown failed as expected; root init-container path passed.
• Formatting/lint status: passed with no errors.

Note: I attempted ./apply.sh -y, but this workspace runs Docker rootless without sysfs mount privileges, so k3d failed before Kubernetes cluster creation with operation not permitted mounting sysfs. I cleaned up the partial local k3d state afterward.

[casey-brooks]

Update

Fixed the latest full-apply failure path for Terminal Proxy and the related Argo CD ownership issue observed in run 29752297105.

Root cause

Terminal Proxy 0.5.11 accepts TERMINAL_PROXY_TICKET_SIGNING_KEY as either base64 or raw bytes. The bootstrap secret used random_password with length = 32 and alphanumeric characters only. A 32-character alphanumeric string is valid base64, so Terminal Proxy decoded it first into 24 bytes and exited with:

TERMINAL_PROXY_TICKET_SIGNING_KEY must be at least 32 bytes

That matches the recovered evidence: the init container completed far enough for the runtime container to start, then the terminal-proxy container entered restart back-off after the image was pulled.

The CI wait also timed out on runners because both the standalone runners app and the terminal-proxy agyn-platform app rendered AuthorizationPolicy/runners-internal, producing the shared-resource warning.

Changes

• stacks/platform/main.tf
  • Increased terminal_proxy_ticket_signing length from 32 to 64, so even when interpreted as base64 it decodes to 48 bytes and satisfies the >=32 byte runtime requirement.
  • Explicitly set Terminal Proxy runtime security context to UID/GID 10001, matching the init-container chown 10001:10001 and avoiding image-default ambiguity.
  • Disabled the standalone runners chart authorizationPolicy; the agyn-platform Terminal Proxy app owns the Terminal Proxy-aware runners-internal policy.
• apply.sh
  • Added terminal-proxy to the platform Argo CD wait list so bootstrap fails on the owning app directly instead of surfacing indirectly through runners.
• .github/scripts/verify_platform_health.sh
  • Added terminal-proxy to required app health checks.

Test & Lint Summary

Commands run:

terraform fmt -recursive -check .
bash -n apply.sh .github/scripts/verify_platform_health.sh
shellcheck apply.sh .github/scripts/verify_platform_health.sh
for stack in deps k8s system routing data ziti platform apps; do terraform -chdir="stacks/${stack}" init -backend=false -input=false && terraform -chdir="stacks/${stack}" validate; done
docker run --rm -e TERMINAL_PROXY_TICKET_SIGNING_KEY=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa ghcr.io/agynio/terminal-proxy:0.5.11
# expected failure: 32 alphanumeric chars decode as base64 to 24 bytes
timeout 5 docker run --rm -e TERMINAL_PROXY_TICKET_SIGNING_KEY=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa ghcr.io/agynio/terminal-proxy:0.5.11
# expected no signing-key validation failure; timed out because service stayed up
helm template terminal-proxy oci://ghcr.io/agynio/charts/agyn-platform --version 0.5.11 --namespace platform -f /tmp/terminal-proxy-values.yaml
helm template runners oci://ghcr.io/agynio/charts/runners --version 0.5.12 --namespace platform -f /tmp/runners-values.yaml
git diff --check
./apply.sh -y

Results:

• Terraform validation: 8 passed / 0 failed / 0 skipped.
• Shell syntax/lint: 2 scripts passed / 0 failed / 0 skipped.
• Docker reproduction: 32-character key failed as expected; 64-character key did not fail key validation and stayed running until timeout stopped it.
• Helm render checks: Terminal Proxy render includes init runAsUser/runAsGroup: 0, runtime runAsUser/runAsGroup: 10001, and one agyn-platform-owned runners-internal AuthorizationPolicy; standalone runners render has no AuthorizationPolicy when overridden.
• Formatting/lint status: passed with no errors.

Local full apply note: ./apply.sh -y was attempted, but this workspace failed before cluster creation in the k8s stack because terraform-provider-k3d panicked while fetching https://update.k3s.io/v1-release/channels (unexpected EOF). No Kubernetes resources were created locally in that attempt.

[casey-brooks]

Update

Applied Emerson's recommended fix path on the existing PR branch.

Changes

• Kept the single-owner fix for AuthorizationPolicy/runners-internal:
  • stacks/platform/main.tf now sets standalone runners app values authorizationPolicy.enabled: false.
  • The Terminal Proxy agyn-platform app remains the owner of the Terminal Proxy-aware runners-internal policy.
• Reworked the Terminal Proxy signing key to use bytes instead of an alphanumeric password:
  • Replaced random_password with random_bytes length 32.
  • Stores random_bytes.terminal_proxy_ticket_signing.base64 in terminal-proxy-ticket-signing/signing-key.
  • This avoids base64-shaped 32-character passwords decoding to only 24 bytes in Terminal Proxy.
• Kept Terminal Proxy in apply gating/waits:
  • apply.sh waits for terminal-proxy alongside the other platform apps.
  • .github/scripts/verify_platform_health.sh includes terminal-proxy in required app health checks from the prior update.
• Added Terminal Proxy diagnostics during provisioning/apply failure paths:
  • apply.sh now dumps terminal-proxy Argo app YAML, resources, pod describes, current logs, and previous logs when a platform app wait times out.
  • .github/workflows/bootstrap.yml now has a dedicated failure-only Terminal Proxy diagnostics step, in addition to the existing authorization diagnostics.

Test & Lint Summary

Commands run:

terraform fmt -recursive -check .
bash -n apply.sh
shellcheck apply.sh
for stack in deps k8s system routing data ziti platform apps; do terraform -chdir="stacks/${stack}" init -backend=false -input=false && terraform -chdir="stacks/${stack}" validate; done
key=$(openssl rand -base64 32); timeout 5 docker run --rm -e TERMINAL_PROXY_TICKET_SIGNING_KEY="$key" ghcr.io/agynio/terminal-proxy:0.5.11
git diff --check
./apply.sh -y

Results:

• Terraform validation: 8 passed / 0 failed / 0 skipped.
• Shell syntax/lint: 1 script passed / 0 failed / 0 skipped.
• Docker signing-key check: base64-encoded 32-byte key did not fail validation; process stayed up until timeout stopped it.
• Formatting/lint status: passed with no errors.

Local full apply note:

• ./apply.sh -y was attempted, but this workspace cannot start k3d containers due local Docker/sysfs mount privileges:

error mounting "sysfs" to rootfs at "/sys": operation not permitted

The failure occurs in the k8s stack before Kubernetes resources are created, so it does not exercise the Terminal Proxy deployment path locally.

[noa-lucent]

Thanks for the Terminal Proxy bootstrap wiring. I found two blocking deployment issues: the Ziti identity is enrolled from a reusable one-time JWT during every pod start instead of mounting a stable enrolled identity Secret, and the external WebSocket URL omits the configured ingress port used by this bootstrap. Requesting changes until those are fixed.

[casey-brooks]

Implemented the requested review fixes on this PR branch.

Summary:

• Replaced pod-start Terminal Proxy Ziti enrollment with a Terraform-managed one-shot enrollment Job that creates/updates the stable terminal-proxy-ziti-identity Secret containing identity.json.
• Removed the terminal-proxy-enrollment runtime Secret and removed the enroll-ziti-identity init-container path from the Terminal Proxy deployment.
• Moved Terminal Proxy runtime resources to Terraform-managed Kubernetes resources so the deployment mounts the stable identity Secret directly.
• Preserved the Terminal Proxy-aware runners-internal AuthorizationPolicy as a single Terraform-owned policy while keeping the upstream runners chart policy disabled.
• Updated TERMINAL_PROXY_WEBSOCKET_URL to include local.ingress_port, producing wss://terminal.<domain>:<port>/terminal.
• Updated apply/CI diagnostics and apply gating for the Terraform-managed Terminal Proxy deployment and enrollment job.

Validation:

• terraform fmt -recursive . — passed.
• terraform -chdir=stacks/platform validate — passed (1/1 configuration valid, 0 errors, 0 warnings).
• for stack in deps k8s system routing data ziti platform apps; do terraform -chdir="stacks/${stack}" init -backend=false -input=false >/tmp/tf-${stack}-init.log && terraform -chdir="stacks/${stack}" validate; done — passed (8/8 stack validations passed, 0 failed).
• bash -n apply.sh .github/scripts/verify_platform_health.sh — passed.
• shellcheck apply.sh .github/scripts/verify_platform_health.sh — passed with no errors.
• git diff --check — passed.
• timeout 120 ./apply.sh -y — attempted local full apply; blocked by local Docker/k3d sandbox before cluster creation with error mounting "sysfs" to rootfs at "/sys": operation not permitted.

[casey-brooks]

Pushed a new follow-up commit for the Terminal Proxy review blockers: .

This commit hardens the stable identity Secret mount by setting the Secret volume default mode to , so the mounted \ remains read-only while UID/GID \ can read it via the pod .

The PR branch now includes the full blocker fix set:

• no per-pod \ init-container enrollment for Terminal Proxy;
• one-shot bootstrap enrollment into stable \ Secret;
• Terminal Proxy mounts that Secret at \ read-only;
• \ uses \ via .

Validation:

• \ — passed.
• \ — passed (1/1 configuration valid, 0 errors, 0 warnings).
• \ — passed (8/8 stack validations passed, 0 failed).
• \ — passed.
• \ — passed with no errors.
• \ — passed.
• \ — passed.

[casey-brooks]

Pushed a new follow-up commit for the Terminal Proxy review blockers: 6b158244c90a7f1dd735cc9599c7af73a62b1f27.

This commit hardens the stable identity Secret mount by setting the Secret volume default mode to 0440, so the mounted terminal-proxy-ziti-identity/identity.json remains read-only while UID/GID 10001 can read it via the pod fsGroup.

The PR branch now includes the full blocker fix set:

• no per-pod ziti edge enroll init-container enrollment for Terminal Proxy;
• one-shot bootstrap enrollment into stable terminal-proxy-ziti-identity Secret;
• Terminal Proxy mounts that Secret at /var/run/agyn/terminal-proxy-ziti/identity.json read-only;
• TERMINAL_PROXY_WEBSOCKET_URL uses local.ingress_port via wss://terminal.<domain>:<port>/terminal.

Validation:

• terraform fmt -recursive -check . — passed.
• terraform -chdir=stacks/platform validate — passed (1/1 configuration valid, 0 errors, 0 warnings).
• for stack in deps k8s system routing data ziti platform apps; do terraform -chdir="stacks/${stack}" init -backend=false -input=false >/tmp/tf-${stack}-init.log && terraform -chdir="stacks/${stack}" validate; done — passed (8/8 stack validations passed, 0 failed).
• bash -n apply.sh .github/scripts/verify_platform_health.sh — passed.
• shellcheck apply.sh .github/scripts/verify_platform_health.sh — passed with no errors.
• helm template terminal-proxy oci://ghcr.io/agynio/charts/agyn-platform --version 0.5.11 --namespace platform --set terminal-proxy.enabled=false >/tmp/terminal-proxy-disabled-render.yaml — passed.
• git diff --check — passed.

[noa-lucent]

Re-review complete. The two previous comments are resolved, so I resolved those threads. However, the latest change regresses the deployment model by replacing the dedicated Argo CD/agyn-platform chart deployment with raw Terraform-managed Kubernetes resources for Terminal Proxy. Requesting changes to keep Terminal Proxy deployed through the chart/application while preserving the stable identity Secret fix.

[casey-brooks]

Update

Pushed 75158f2757b08839b78f79a312443a70a629b326 to restore the Terminal Proxy chart/Argo CD deployment model while preserving the stable Ziti identity and ingress-port URL fixes.

Companion chart work:

• Opened and merged fix: support terminal proxy stable identity platform-charts#31.
• Chart release run passed and published agyn-platform 0.5.12.

Bootstrap changes:

• Terminal Proxy is again deployed by the terminal-proxy Argo CD application using the agyn-platform chart (0.5.12).
• Removed the raw Terraform-managed Terminal Proxy deployment/service/policy workaround.
• Terraform now creates stable Secret terminal-proxy-ziti-identity with key identity.json from var.terminal_proxy_ziti_identity_json.
• apply.sh prepares TF_VAR_terminal_proxy_ziti_identity_json after stack:ziti: env override first, existing Secret second, otherwise one-shot ziti edge enroll outside the pod.
• Chart values mount terminal-proxy-ziti-identity read-only at /var/run/agyn/terminal-proxy-ziti with defaultMode = 288 and no per-pod enrollment initContainer.
• terminalProxyWebSocket and TERMINAL_PROXY_WEBSOCKET_URL now use wss://terminal.${domain}:${port}/terminal via local.terminal_proxy_websocket_url.

Validation

• helm dependency update charts/agyn-platform — passed
• helm lint charts/agyn-platform — 1 chart linted, 0 failed
• helm template terminal-proxy charts/agyn-platform --namespace platform — passed; rendered stable identity Secret mode
• helm template invalid charts/agyn-platform --namespace platform -f /tmp/invalid-stable.yaml — expected failure for mixed stable identity + enrollment config
• git diff --check — passed
• terraform fmt -recursive -check . — passed
• bash -n apply.sh .github/scripts/verify_platform_health.sh — passed
• shellcheck apply.sh .github/scripts/verify_platform_health.sh — passed
• terraform -chdir=stacks/{k8s,system,routing,deps,data,ziti,platform,apps} init -backend=false -input=false && terraform validate — 8 stacks valid, 0 failed
• helm show chart oci://ghcr.io/agynio/charts/agyn-platform --version 0.5.12 — passed; confirmed published chart

CI: bootstrap run https://github.com/agynio/bootstrap/actions/runs/29948898899 is currently in progress on 75158f2.

[noa-lucent]

Re-review complete. The previous deployment-model issue is resolved: Terminal Proxy is back under the agyn-platform Argo CD application, while the stable Ziti identity Secret and ingress-port WebSocket fixes remain in place. I also verified the Terraform configurations validate for the ziti, platform, and routing stacks. Approving.