fix(ci): make the visual harness truthful - build drafts, allow first baselines - #417
Merged
Merged
Conversation
… baselines The first CI baseline-record run (30629929407) failed with 4 test failures and never committed baselines. Two root causes: 1. CI test builds lacked --buildDrafts: the codeblock-styles fixture is a draft post, so every CI visual run (including the new PR gate) 404'd it and failed both codeblock tests - noisy-by-construction. test.yml now sets BUILD_DRAFTS='1' on its setup-hugo step (bin/hugo-build already reads it; matches local test builds). 2. snap_diff fail_if_new hard-errors on missing baselines under ENV["CI"], so record mode could never create a FIRST baseline for pages added since the last recording (vibe_code_rescue had no linux/ baseline). Record mode (FORCE_SCREENSHOT_UPDATE=true) now disables fail_if_new alongside fail_on_difference. Both gotchas recorded in .okf/build/ci-gates.md. After merge: re-dispatch Screenshot Tests with update-baselines=true to land the pinned-stack linux/ baselines (closes the pending re-baseline). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXHeUErqoiyH9xN1mjC8cH
Contributor
|
Warning Review limit reached
Next review available in: 49 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe screenshot workflow now includes Hugo drafts. Forced screenshot updates now allow missing baselines. CI gate documentation and the visual-gate log record both changes. ChangesScreenshot CI gate corrections
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/test.yml:
- Around line 64-67: Replace or supplement the unsupported incident assertions
with durable citations: in .github/workflows/test.yml lines 64-67, cite the
draft-fixture 404; in .okf/build/ci-gates.md lines 52-55, cite both root causes;
and in .okf/log.md lines 5-12, cite the failures and Linux re-record. Use one
shared CI run, issue, or verification artifact where it supports all claims.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: a63136cc-3de2-4ed5-b7ea-0982e41b220a
📒 Files selected for processing (4)
.github/workflows/test.yml.okf/build/ci-gates.md.okf/log.mdtest/support/setup_snap_diff.rb
…eedback) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXHeUErqoiyH9xN1mjC8cH
The paths filter missed test/support/** (snap_diff/capybara config) and the workflow file itself, so this PR's harness changes never triggered the visual job - the gate could not test itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXHeUErqoiyH9xN1mjC8cH
Contributor
Screenshot diffs detected
|
pftg
pushed a commit
that referenced
this pull request
Jul 31, 2026
bin/README.md listed four scripts that do not exist (debug, benchmark-dev, bunx, generate-cover-image) and omitted eleven that do (qtest, dtest-all, setup-test-env, build-if-stale, hugo-build, ...). Rewritten to match the actual directory contents. .okf test-gates: drop the stale 'CI does NOT run screenshot diffs' claim (superseded by #413/#417), record the shared build-if-stale warm-tree flow, qtest untracked-file detection, and the .gitignore dot-path gotcha that had kept the pre-push hook out of the repo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXHeUErqoiyH9xN1mjC8cH
pftg
added a commit
that referenced
this pull request
Jul 31, 2026
* fix(devx): R3-1 - one truthful build path for every test runner All four runners (test/qtest/dtest/dtest-all) now build through a shared bin/build-if-stale helper that routes via bin/hugo-build (PurgeCSS cold-start warm-up guard) and skips the rebuild on a warm tree: - bin/test: DEST honors a caller-preset HUGO_DEFAULT_PATH and the build is skipped entirely under PRECOMPILED_ASSETS - un-breaks bin/dtest, whose container has no hugo and points at _dest/public-dtest - bin/qtest: shares bin/test's _dest/public-test tree (warm reruns skip the ~26s rebuild it previously paid on every micro-commit), and --changed now counts untracked files (a brand-new pages/foo.css no longer exits 0 green with 'nothing to run') - staleness list gains assets/ static/ package.json bun.lockb - .githooks/pre-push: run each unit-test file in its own ruby invocation (ruby file1 file2 executes only file1) - and commit the hook itself: the .* gitignore rule had kept it out of the repo, so fresh clones got a hooksPath pointing at nothing (added !.githooks) - bin_scripts_test: 3 new guards - no bare 'hugo --environment production' outside hugo-build/build-if-stale, pre-push multi-file ruby anti-pattern, bin/test HUGO_DEFAULT_PATH contract Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXHeUErqoiyH9xN1mjC8cH * docs(devx): R3-1 - truthful bin/README, test-gates concept + log entry bin/README.md listed four scripts that do not exist (debug, benchmark-dev, bunx, generate-cover-image) and omitted eleven that do (qtest, dtest-all, setup-test-env, build-if-stale, hugo-build, ...). Rewritten to match the actual directory contents. .okf test-gates: drop the stale 'CI does NOT run screenshot diffs' claim (superseded by #413/#417), record the shared build-if-stale warm-tree flow, qtest untracked-file detection, and the .gitignore dot-path gotcha that had kept the pre-push hook out of the repo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXHeUErqoiyH9xN1mjC8cH * fix(devx): address review - rake guards entrypoint + deletion staleness - New `rake test:guards` task (toolchain pins + bin scripts, ~2s) is the pre-push hook's single test entrypoint - repo-provided command instead of direct ruby file invocations, and structurally immune to the multi-file-ruby trap - bin/build-if-stale staleness probe now includes directory mtimes: deleting a source file leaves nothing to be -newer, but bumps its parent dir - verified warm tree goes stale on a removed content file - pre-push guard test asserts the rake entrypoint is present (no longer passes vacuously if the hook drops its test invocation) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXHeUErqoiyH9xN1mjC8cH --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The next DevX improvement per the approved plan (YAGNI-scoped: 3 files): make the new CI visual harness tell the truth. The first baseline-record dispatch (run 30629929407) failed — 104 screenshots compared clean, but 4 test failures blocked the commit step. Both root causes diagnosed from the job log and fixed:
1. CI test builds had no draft fixtures
test_codeblock_language_styles(desktop+mobile) failed withfound 1 match: "File not found: /blog/codeblock-styles-fixture/"— the fixture is a draft post. Local runs build with--buildDrafts;bin/hugo-buildonly passes it whenBUILD_DRAFTSis set, and nothing in CI set it. This made the new PR report-only gate noisy-by-construction (would fail on every PR).test.yml's setup-hugo step now setsBUILD_DRAFTS: '1'.2. Record mode couldn't create first baselines
test_vibe_code_rescue(desktop+mobile) failed withNo existing screenshot found for .../linux/desktop/vibe_code_rescue.base.png— snap_diff'sfail_if_newhard-errors underENV["CI"], and record mode only disabledfail_on_difference. Pages added after the last recording could never get a first CI-recorded baseline.setup_snap_diff.rb's record-mode block now disablesfail_if_newtoo.Both gotchas are recorded in
.okf/build/ci-gates.mdso nobody re-diagnoses them.Verification
test.yml→ the paths filter fires the visual PR job on this very PR; it coming back green proves fix 1 end-to-end (fixture found, codeblock tests pass) with compare-mode semantics unchanged.ruby -c+ YAML parse clean.Screenshot Testswithupdate-baselines: true— expected: green run + achore: update screenshot baselines [ci skip]commit with the pinned-stacklinux/PNGs including NEWvibe_code_rescuebaselines. That closes the last open re-baseline item.Deferred (YAGNI, per plan): CI artifact-sharing, parallelization, blocking-mode flip (stays report-only until a clean soak week on real PRs).
🤖 Generated with Claude Code
https://claude.ai/code/session_01PXHeUErqoiyH9xN1mjC8cH
Generated by Claude Code
Summary by CodeRabbit
Bug Fixes
Documentation