Skip to content

Sweep leaked workloads after local e2e runs - #6367

Merged
jhrozek merged 1 commit into
mainfrom
worktree-e2e-cleanup-sweep
Aug 19, 2026
Merged

Sweep leaked workloads after local e2e runs#6367
jhrozek merged 1 commit into
mainfrom
worktree-e2e-cleanup-sweep

Conversation

@jhrozek

@jhrozek jhrozek commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Local task test-e2e runs the whole test/e2e Ginkgo suite unfiltered with a single 20m --timeout, while CI splits the same suite into ~15 label-filtered jobs (some needing up to 25m for one slice alone). A full local run regularly exceeds the 20m budget.
  • When Ginkgo's timeout fires, it force-kills the process after a 30s grace period, skipping any AfterEach/AfterSuite cleanup that hadn't finished yet. Every thv workload (and its Docker network) created up to that point stays behind. Enough of these accumulating over time exhausted the local Docker network address pool and broke unrelated, real MCP workloads on a dev machine.
  • Per-test cleanup code itself is not missing or broken — it's just skipped when the whole process dies. This adds a shell-level safety net in test/e2e/run_tests.sh that survives that failure mode: it snapshots thv list --all before ginkgo run and diffs it against the same listing afterward, removing only workloads that appeared during the run. That's deliberately not a hardcoded name-prefix list — test files use dozens of ad hoc naming schemes and there's no existing "e2e"/"test" label to key on — so a before/after diff is the only approach that can't misfire on a workload a developer already had running.
  • The sweep runs unconditionally (pass, fail, or timeout-kill) and doesn't change the script's exit code.

Type of change

  • Bug fix

Test plan

  • Manual testing (describe below)

Manual verification only, no full e2e run:

  • bash -n test/e2e/run_tests.sh — syntax clean
  • shellcheck — no new blocking findings
  • Read through cmd/thv/app/list.go and rm.go to confirm thv list --all --format json and thv rm <name> behave as the script assumes

Does this introduce a user-facing change?

No — this only affects local developer tooling for running the e2e suite.

Special notes for reviewers

  • test/e2e/run_tests.bat (Windows) is not touched. The diff-based sweep needs comm/process substitution with no clean batch equivalent; mirroring this for Windows would need a different implementation.
  • This does not fix the underlying cause of local runs legitimately exceeding the 20m timeout (no label-splitting locally, unlike CI). That's a separate follow-up — this PR only adds the safety net so a timeout-kill stops leaking Docker resources in the meantime.

A full unfiltered `task test-e2e` regularly exceeds Ginkgo's 20m
--timeout, since CI splits the same suite into ~15 label-filtered
jobs. When the timeout fires, Ginkgo hard-kills the process after a
30s grace period, skipping any AfterEach/AfterSuite cleanup that
hadn't finished. Workloads (and their Docker networks) created up to
that point are then orphaned, and enough of them can exhaust Docker's
network pool and break unrelated real MCP workloads on the machine.

Snapshot `thv list --all` before the ginkgo run and diff it against
the same listing afterward, then remove only the workloads that
appeared during the run. This targets exactly what leaked without
touching any workload a developer already had running, and needs no
knowledge of the many ad hoc naming schemes test files use to name
their workloads. The sweep runs unconditionally after ginkgo exits
(pass, fail, or timeout-kill) and never changes the script's exit
code.
@github-actions github-actions Bot added the size/XS Extra small PR: < 100 lines changed label Aug 18, 2026
@codecov

codecov Bot commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.02%. Comparing base (7a0c0a5) to head (e2b503e).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6367      +/-   ##
==========================================
+ Coverage   72.97%   73.02%   +0.04%     
==========================================
  Files         742      742              
  Lines       78398    78489      +91     
==========================================
+ Hits        57208    57313     +105     
+ Misses      17201    17189      -12     
+ Partials     3989     3987       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jhrozek
jhrozek merged commit cb0359d into main Aug 19, 2026
45 checks passed
@jhrozek
jhrozek deleted the worktree-e2e-cleanup-sweep branch August 19, 2026 06:56
@github-actions github-actions Bot mentioned this pull request Aug 26, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XS Extra small PR: < 100 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants