Skip to content

test/bench-ablation.test.ts flakes under load and has now failed CI twice #88

Description

@MongLong0214

runner: ablation conditions > accepts all six arms has failed CI on two separate PRs (#79, #87) and fails locally under concurrent load. It passes in isolation every time.

Why this is worth fixing rather than re-running

A flaky test in a required check trains everyone — including me, twice today — to re-run without reading the failure. That is the same habit that let CI stay red for eight commits earlier in this project while being reported green five times.

It also costs a full CI cycle each time, and it lands on whichever PR happens to be in flight rather than on the change that would have caused a real failure.

What is known

  • Passes alone: npx vitest run test/bench-ablation.test.ts → 20/20
  • Fails under load: locally with several concurrent processes, and on GitHub's runners
  • The failing assertion is in the six-arm condition test; the error surfaces as Command failed: from a spawned node process, which suggests a timeout or a resource limit rather than a logic error

Direction

Establish which it is before changing anything — a timeout raised to hide a real contention bug is worse than the flake. If it is a timeout, the test spawns real processes and its budget should scale with available parallelism rather than being a fixed 5000 ms. test/mcp.test.ts has the same 5000 ms timeout symptom under load and may share a cause; check both together.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions