Sweep leaked workloads after local e2e runs - #6367
Merged
Merged
Conversation
A full unfiltered `task test-e2e` regularly exceeds Ginkgo's 20m --timeout, since CI splits the same suite into ~15 label-filtered jobs. When the timeout fires, Ginkgo hard-kills the process after a 30s grace period, skipping any AfterEach/AfterSuite cleanup that hadn't finished. Workloads (and their Docker networks) created up to that point are then orphaned, and enough of them can exhaust Docker's network pool and break unrelated real MCP workloads on the machine. Snapshot `thv list --all` before the ginkgo run and diff it against the same listing afterward, then remove only the workloads that appeared during the run. This targets exactly what leaked without touching any workload a developer already had running, and needs no knowledge of the many ad hoc naming schemes test files use to name their workloads. The sweep runs unconditionally after ginkgo exits (pass, fail, or timeout-kill) and never changes the script's exit code.
jhrozek
requested review from
ChrisJBurns,
JAORMX,
amirejaz,
aponcedeleonch,
rdimitrov and
reyortiz3
as code owners
August 18, 2026 20:41
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6367 +/- ##
==========================================
+ Coverage 72.97% 73.02% +0.04%
==========================================
Files 742 742
Lines 78398 78489 +91
==========================================
+ Hits 57208 57313 +105
+ Misses 17201 17189 -12
+ Partials 3989 3987 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ChrisJBurns
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
task test-e2eruns the wholetest/e2eGinkgo suite unfiltered with a single 20m--timeout, while CI splits the same suite into ~15 label-filtered jobs (some needing up to 25m for one slice alone). A full local run regularly exceeds the 20m budget.AfterEach/AfterSuitecleanup that hadn't finished yet. Everythvworkload (and its Docker network) created up to that point stays behind. Enough of these accumulating over time exhausted the local Docker network address pool and broke unrelated, real MCP workloads on a dev machine.test/e2e/run_tests.shthat survives that failure mode: it snapshotsthv list --allbeforeginkgo runand diffs it against the same listing afterward, removing only workloads that appeared during the run. That's deliberately not a hardcoded name-prefix list — test files use dozens of ad hoc naming schemes and there's no existing "e2e"/"test" label to key on — so a before/after diff is the only approach that can't misfire on a workload a developer already had running.Type of change
Test plan
Manual verification only, no full e2e run:
bash -n test/e2e/run_tests.sh— syntax cleanshellcheck— no new blocking findingscmd/thv/app/list.goandrm.goto confirmthv list --all --format jsonandthv rm <name>behave as the script assumesDoes this introduce a user-facing change?
No — this only affects local developer tooling for running the e2e suite.
Special notes for reviewers
test/e2e/run_tests.bat(Windows) is not touched. The diff-based sweep needscomm/process substitution with no clean batch equivalent; mirroring this for Windows would need a different implementation.