fix: stop y-websocket connection leaks in shared editing and reap idle /rtc connections at the gateway

[aicam]

What changes were proposed in this PR?

Fixes two distinct y-websocket (/rtc) connection leaks in the shared-editing code path, plus a gateway-side idle-connection reaper as defense-in-depth. Both leaks live on SharedModel → WebsocketProvider, and together they let the number of open realtime connections grow far out of proportion to the number of users until the shared-editing connection cap is exhausted and collaborative cursors/awareness stop syncing.

Leak B — throwaway validation graph opens a socket per edit (dominant).
On every operator/link/property edit, WorkflowCompilingService builds a throwaway WorkflowGraph via ValidationWorkflowService.getValidTexeraGraph() just to produce a logical plan for compilation. WorkflowGraph's constructor unconditionally created a SharedModel, which unconditionally opened a new auto-connecting /rtc WebSocket in a random room. That graph is read once and never destroyed, so each edit leaked a socket that reconnects forever. A single user editing a workflow accumulates dozens of permanent sockets.

Fix: SharedModel now opens a /rtc connection only when a workflow ID (wid) is provided — the presence of a wid is already the contract for joining shared editing (loadNewYModel(workflowId, ...)). With no wid, the model stays purely local: no WebsocketProvider is created and a standalone Awareness keeps downstream reads of awareness/clientId working. Throwaway validation/compilation graphs, unit-test fixtures, and new unsaved canvases have no wid, so they never touch the network — by construction, with no extra configuration flag. (The previous random-room fallback provided no real collaboration anyway — each client got its own random room — so removing it changes no user-visible behavior.)

Leak A — incomplete provider teardown (per-navigation).
SharedModel.destroy() called wsProvider.disconnect() (and only when the socket happened to be OPEN). In y-websocket@1.4.0, disconnect() does not clear the provider's _checkInterval/_resyncInterval reconnect timers, so the provider is never garbage-collected and, if it was mid-reconnect at destroy time (common during workflow switches), keeps re-opening a zombie socket for the life of the tab.

Fix: call wsProvider.destroy() unconditionally, which clears those timers, removes the beforeunload handler, broadcasts awareness removal (peers drop our cursor), and closes the socket.

Behavior for the real collaborative graph is unchanged: once a workflow is loaded with its wid, it constructs a WebsocketProvider exactly as before. wsProvider is now optional (WebsocketProvider | undefined), so its remaining access sites use ?..

Gateway change — reap idle /rtc connections (defense-in-depth).
The y-websocket reference server is single-process and every browser tab holds a long-lived WebSocket, so abandoned/leaked realtime sockets can pile up against the gateway's per-cluster max_connections circuit breaker (Envoy's default is 1024) and starve legitimate co-editing traffic. This PR gives the /rtc path its own HTTPRoute (so a policy can target just that cluster without affecting the other static routes) and attaches an Envoy Gateway BackendTrafficPolicy that:

• drops upstream connections idle for longer than connectionIdleTimeout (default 5m), and
• raises the per-cluster circuitBreaker.maxConnections (default 8192) for headroom.

Both knobs are configurable under yWebsocketServer.trafficPolicy in values.yaml; setting trafficPolicy to null omits the policy. The policy targets only the shared-editing HTTPRoute via targetRefs, so no other backend's limits change. This is complementary to the frontend fix — the frontend fix stops creating the leaked sockets; the gateway policy caps the blast radius of any that still linger.

Note — one extra housekeeping edit: since BackendTrafficPolicy is an Envoy Gateway resource, both traffic-policy templates now live under templates/base/gateway/ alongside the other gateway resources. This includes moving the pre-existing agent-service-backend-traffic-policy.yaml out of templates/base/agent-service/ — a pure file move (Helm renders templates recursively), included here so all gateway-layer templates sit in one folder.

Any related issues, documentation, discussions?

Fixes #6239.

How was this PR tested?

Existing frontend unit tests continue to pass; the mock-workflow-plan.ts fixtures construct their graphs without a wid, so tests open no real /rtc sockets.

Manual verification, client-side (browser DevTools → Network → WS filter):

Before the fix (reproducing the leak):

1. Open a workflow → one /rtc WebSocket.
2. Each operator add/delete, property edit, or link change spawned an additional /rtc socket to a random room that never closed and auto-reconnected forever — the WS count climbed monotonically with every edit.
3. Server-side, the same growth was observed as a steadily climbing count of ESTABLISHED connections on the y-websocket server (hundreds of connections from a handful of users, dominated by random-room sockets), which is what exhausted Envoy's default 1024 per-cluster connection cap and broke shared editing in production.

After the fix:

1. Open a workflow → exactly one /rtc WebSocket (unchanged).
2. Add/delete operators, edit properties, add/remove links repeatedly → the count stays at one.
3. Switch workflow A → B → the old socket closes and one new socket opens (count stays 1).
4. Navigate to the dashboard / close the tab → the socket closes (count → 0), with no lingering reconnects.
5. Server-side, the established-connection count stays flat (≈ number of open collaborative tabs) and drops when tabs close, instead of climbing with every edit.

Gateway change: helm template was run to confirm the chart renders valid manifests — the /rtc HTTPRoute and the BackendTrafficPolicy (with connectionIdleTimeout and circuitBreaker.maxConnections) are emitted with the policy correctly targeting only the shared-editing route, and the policy is omitted when yWebsocketServer.trafficPolicy is null.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @bobbai00, @aglinxinyuan
  You can notify them by mentioning @bobbai00, @aglinxinyuan in a comment.

[codecov-commenter]

⚠️ JUnit XML file not found

The CLI was unable to find any JUnit XML files to upload.
For more help, visit our troubleshooting guide.

[Copilot]

Pull request overview

This PR addresses runaway /rtc (y-websocket) WebSocket growth caused by frontend shared-editing objects being created without proper teardown (and, in one path, created unintentionally for throwaway validation graphs). It also adds an Envoy Gateway policy as defense-in-depth to reap idle /rtc upstream connections and raise the connection ceiling for that cluster.

Changes:

• Added an enableSharedEditing switch so throwaway/validation WorkflowGraph instances do not create a WebsocketProvider (avoids per-edit leaked /rtc sockets).
• Fixed shared-editing teardown by destroying the provider (wsProvider.destroy()) rather than only disconnecting, eliminating reconnect-timer “zombie” providers.
• Split /rtc into its own Gateway HTTPRoute and added an optional BackendTrafficPolicy (idle timeout + increased maxConnections) configurable via Helm values.

Reviewed changes

Copilot reviewed 5 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
frontend/src/app/workspace/service/workflow-graph/model/shared-model.ts	Makes wsProvider optional, supports local-only mode, and ensures provider is fully destroyed on teardown.
frontend/src/app/workspace/service/workflow-graph/model/workflow-graph.ts	Threads enableSharedEditing into SharedModel construction to avoid unintended network connections.
frontend/src/app/workspace/service/validation/validation-workflow.service.ts	Builds validation graphs with enableSharedEditing=false to prevent per-edit WebSocket leaks.
frontend/src/app/workspace/service/workflow-graph/model/workflow-action.service.ts	Updates provider access for optionality (but one call site still needs full null-safe access).
frontend/src/app/workspace/service/execute-workflow/mock-workflow-plan.ts	Updates fixtures to create local-only graphs (no sockets during tests/fixtures).
bin/k8s/templates/base/gateway/gateway-routes.yaml	Moves /rtc to its own HTTPRoute so policies can target it without affecting other routes.
bin/k8s/templates/base/shared-editing-server/shared-editing-server-backend-traffic-policy.yaml	Adds optional Envoy Gateway BackendTrafficPolicy for /rtc (idle reaping + circuit breaker headroom).
bin/k8s/values.yaml	Adds yWebsocketServer.trafficPolicy knobs (connectionIdleTimeout, maxConnections).
bin/k8s/values-development.yaml	Mirrors the new yWebsocketServer.trafficPolicy knobs for dev values.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[aglinxinyuan]

You manually verified the behavior after the fix. Could you also verify it before the fix? Specifically, we'd like to confirm that there were consistently multiple WebSocket connections before this change.

Also, there's no need to list the files changed in the PR description.

[Yicong-Huang]

do we need to backport the fix to v1.2?

[Yicong-Huang]

why do we need a enableSharedEditing config? Please find a better way to pass the config. Bundling it with WorkflowGraph is not ideal.

[xuang7]

Could you open a separate PR targeting the release/v1.2 branch for this fix? Thanks.

[aicam]

@aglinxinyuan Good point — I verified the before-fix behavior as well and added it to the description ("How was this PR tested?"). Summary of the before-fix evidence:

Client-side (DevTools → Network → WS, pre-fix build): opening a workflow starts one /rtc socket, and then every single edit (operator add/delete, property change, link add/remove) spawns an additional /rtc socket to a random-UUID room. These extra sockets never close and auto-reconnect forever, so the WS count climbs monotonically with every edit — a few minutes of editing accumulates dozens of permanent connections from one tab.

Server-side (production y-websocket server): counting ESTABLISHED connections on the shared-editing server showed the same signature — hundreds of connections from only a handful of active users, steadily climbing while they edited, with the room names dominated by random UUIDs (the throwaway validation graphs) rather than numeric workflow IDs. That growth is what exhausted Envoy's default per-cluster limit of 1024 connections and broke shared editing in production (#6239). Tabs still running the pre-fix bundle keep leaking in this pattern today, while tabs on the fixed bundle hold exactly one connection per open workflow.

Also trimmed the changed-files list from the PR description as suggested.

[aicam]

@Yicong-Huang Both points addressed:

enableSharedEditing config: removed entirely in 3ae20d7. Instead of a flag, SharedModel now connects lazily — it opens a /rtc socket only when a workflow ID is provided, which is the contract loadNewYModel(workflowId, ...) already documented. Graphs without a wid (throwaway validation/compilation graphs, test fixtures, unsaved canvases) are local-only by construction, so WorkflowGraph's constructor is back to its original 3-argument signature and carries no shared-editing configuration. PR description updated to match.

Backport to v1.2: I'd say yes for the frontend commit — the per-edit leak exists in v1.2 as well, and any deployment fronted by a connection-capped proxy will eventually hit it. The frontend change is small and self-contained, so it should cherry-pick cleanly. The k8s/gateway part is optional for the backport (it only applies to Envoy Gateway deployments); happy to open the backport PR once this merges if you'd like.

[aicam]

You manually verified the behavior after the fix. Could you also verify it before the fix? Specifically, we'd like to confirm that there were consistently multiple WebSocket connections before this change.

Also, there's no need to list the files changed in the PR description.

I don't have screenshot but we experienced disconnected users when they used shared editing and after checking the logs, we realized that we had over 800 websocket connections for 6 shared editing sessions

[Copilot]

Pull request overview

Copilot reviewed 5 out of 11 changed files in this pull request and generated no new comments.

[aglinxinyuan]

The logic looks correct to me. I don't have a way to verify that this actually fixes the issue since I can't reproduce it. However, as long as this change doesn't break anything, I'm okay with it.

[aicam]

During final testing of this PR, I found another issue that is not relevant to this one but still happen in shared editing feature. To make sure fixes are done properly, I will ask @Xiao-zhen-Liu. I close this PR for now, we do not consider it for RC v1.2.