Three independent agents hit the same wall this week while verifying unrelated PRs (#434-#438 batch); all three reproduce on pristine builds of their base commits, so none are regressions — this is the tier-1 script itself.
test_count_baseline is ~350 behind. scripts/eval/tier1-manifest.json says 656; the suite is at 1000+. The ratchet has not ratcheted in a while — it should be bumped at each release cut (and arguably asserted against drift in CI so it can't silently stall again).- The
tests check cannot read a count off a cached run. On zig 0.17.0-dev.813, a fully-cached zig build test does not print the N/N tests passed summary line the script's comment assumes; the check reports "could not read the test count out of the build summary" and FAILs on a build that is actually green. Workaround in the field: rm -rf .zig-cache/h and re-run — but the script should handle the cached shape itself (e.g. run the compiled test artifact directly, which always prints All N tests passed).
- The
reach check can mis-fire on filtered builds: if a -Dtest-filter build is the newest artifact in .zig-cache/o, reachability counts that filtered binary and reports missing tests that exist.
All three degrade trust in a gate that's otherwise doing its job — the pre-push hook caught real regressions in the past, and false FAILs teach people to ignore it.
Three independent agents hit the same wall this week while verifying unrelated PRs (#434-#438 batch); all three reproduce on pristine builds of their base commits, so none are regressions — this is the tier-1 script itself.
test_count_baselineis ~350 behind.scripts/eval/tier1-manifest.jsonsays656; the suite is at 1000+. The ratchet has not ratcheted in a while — it should be bumped at each release cut (and arguably asserted against drift in CI so it can't silently stall again).testscheck cannot read a count off a cached run. On zig 0.17.0-dev.813, a fully-cachedzig build testdoes not print theN/N tests passedsummary line the script's comment assumes; the check reports "could not read the test count out of the build summary" and FAILs on a build that is actually green. Workaround in the field:rm -rf .zig-cache/hand re-run — but the script should handle the cached shape itself (e.g. run the compiled test artifact directly, which always printsAll N tests passed).reachcheck can mis-fire on filtered builds: if a-Dtest-filterbuild is the newest artifact in.zig-cache/o, reachability counts that filtered binary and reports missing tests that exist.All three degrade trust in a gate that's otherwise doing its job — the pre-push hook caught real regressions in the past, and false FAILs teach people to ignore it.