bench(public): regenerate the Node/Bun baseline on the mac mini (v0.5.1355) - #7641
Merged
Conversation
added 2 commits
August 8, 2026 13:46
….1355) The dev Mac (M1 Max) cannot reach the harness's CPU-quiet gate for the next ~14 hours, so this run moves to the pinned mac mini at the owner's direction. HOST CHANGE, STATED PLAINLY: host.cpu goes Apple M1 Max / 64 GB -> Apple M1 / 8 GB. This STEPS the published series -- rows are not comparable to the 2026-08-07 artifact, and the absolute numbers are a slower, smaller-memory machine. The relative Perry/node/bun ordering is what carries over, and the generated table records the host, so a reader can see it. Three failures preceded this run and all three were mechanical, not measurement: 1. hyperfine/gtimeout/esbuild were absent on the mini. My memory names five undeclared harness deps; I verified node/bun/zig and stopped at three. Shipped all three (checked they carry no Homebrew dylib deps first). 2. The harness refuses to measure a dirty tree, and a prior aborted run's generated RESULTS files are exactly what dirties it. 3. The harness writes hyperfine exports to FIXED /tmp paths and the write does not truncate: stale exports from an Aug 6 session left a trailing document, so a shorter new write produced two JSON documents in one file and the reader died with "Extra data". (2) and (3) are handled in the launcher, deliberately NOT in run.sh -- that file is fingerprinted, and editing it would invalidate the artifact the run exists to produce. public_baseline.py check exits 0.
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Aug 8, 2026
proggeramlug
pushed a commit
that referenced
this pull request
Aug 31, 2026
…setup (#9277) * fix(bench): reject implausible wall times instead of publishing them honest_bench can record a wall time that is physically impossible and nothing downstream notices. The artifact at 38ff7ec holds 150 negative wall_ms samples out of 300, and REPORT.md is generated from them, so every timing cell in that file currently reads 0.0 ms. The runs themselves are fine (exit_code 0, checksums match); only the timing is meaningless. report.py filters on exit_code == 0 alone, so those samples reach statistics.median() and render as confident results. Add two guards: - run_bench.sh aborts, printing the offending start_ns/end_ns pair, rather than writing a non-positive sample. - report.py refuses to build a report from an artifact containing non-positive successful samples, and says how many. Verified against the current committed results.json (exits 1) and against the pre-#7641 artifact at 7beb3a5 (regenerates normally). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(bench): correct three setup issues in polyglot bench.rs Each of these is already described in benchmarks/polyglot/RESULTS.md. bench_array_write: suite/03_array_write.ts fills every slot before calling Date.now(), so its timed loop overwrites resident pages. The Rust version used vec![0.0; 10_000_000] -- calloc, lazily mapped -- and timed the loop that first-touches them, paying ~10M page faults the TS loop had already paid. Pre-touch before starting the timer. 19 ms -> 5 ms locally. fib: was i32. RESULTS.md states that Perry's inference refines the TS number parameter to i64, so i64 is the like-for-like peer. 240 ms -> 214 ms. bench_object_create: without a barrier LLVM proves Point never escapes and deletes the loop, which is why the row reports 0 ms. black_box makes it measure the allocation it claims to. 0 ms -> 1 ms. These move published numbers, so the polyglot sweep needs a re-run on the project's reference hardware. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(readme): pin the stale REPORT.md citation, add the 500k JSON row The convolution and JSON values in the performance table match the artifact at 7beb3a5 exactly, so they were correct when written -- but #7641 regenerated that file with an invalid clock, and the copy at HEAD no longer contains them. Pin the citation to the revision that does, and note it should be re-pinned to main after the next good regeneration. Also add the 500k-record JSON pipeline row, using the figures already in REPORT.md's own prose (Perry 1,649 / Rust 604 / Node 1,010 / Bun 647). The table showed only the 100-record fixture, and the paragraph directly beneath it says the project publishes the workloads where the JITs win. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Regenerates the public artifact and unblocks
lint's freshness gate, which has been red since #7601 editedjson_pipeline.ts— a fingerprinted harness file.Host change, stated up front
host.cpugoes Apple M1 Max / 64 GB → Apple M1 / 8 GB. The dev Mac cannot reach the harness's CPU-quiet gate for the next ~14 hours, so this run moved to the pinned mini at the owner's direction.This steps the published series. Rows are not comparable to the 2026-08-07 artifact — these are a slower, smaller-memory machine. What carries over is the relative Perry/node/bun ordering, and the generated table records the host so a reader can see which machine produced it. The suite table moves accordingly (e.g.
factorial96→94 ms,prime_sieve30→28 ms, andfactorialflipsmixed → win vs bothon a 1 ms margin).python3 benchmarks/public_baseline.py checkexits 0.Three mechanical failures preceded this, none of them measurement
Recording them because each is a trap the next person will hit:
hyperfine,gtimeout,esbuildwere absent on the mini. These are undeclared harness deps — the harness gates on node/bun versions but dies without these. I verified node/bun/zig and stopped at three of five. Shipped all three after checking they carry no Homebrew dylib dependencies.RESULTS.mdfiles are exactly what dirties it./tmppaths, and the write does not truncate. Stale exports from an Aug 6 session meant a shorter new document landed on a longer old one, leaving a trailing]}— two JSON documents in one file, and the reader died withJSONDecodeError: Extra data. Confirmed by reading the duplicate closing sequence at lines 183–192.(2) and (3) are now handled in the launcher, deliberately not in
run.sh— that file is inHARNESS_PATHS, so editing it would invalidate the very artifact the run produces.Note for the next regeneration
The zig-link trap recorded in my notes (
__availability_version_checkfailing on the mini) is stale — both honest_bench zig workloads built cleanly there with the pinned 0.15.2. The mini is a viable baseline host provided the five undeclared deps are present.