Skip to content

fix(testing): add ignored-test audit script + weekly cron + policy (#839)#881

Merged
zackees merged 1 commit into
mainfrom
fix/839-ignored-test-audit-script-cron-policy
Jun 30, 2026
Merged

fix(testing): add ignored-test audit script + weekly cron + policy (#839)#881
zackees merged 1 commit into
mainfrom
fix/839-ignored-test-audit-script-cron-policy

Conversation

@zackees

@zackees zackees commented Jun 30, 2026

Copy link
Copy Markdown
Member

Summary

Adds visibility for #[ignore]-gated tests across the workspace:

  • ci/audit_ignored_tests.py — stdlib-only Python script that walks the workspace, parses every #[ignore] / #[ignore = \"...\"] attribute, and emits either CSV (default) or markdown. Example CSV row:
    crates/fbuild-deploy/src/esp32/tests.rs,try_verify_deployment_real_esp32,requires real ESP32 board — set ESP32_PORT and ESP32_FIRMWARE,714
    
  • .github/workflows/audit-ignored-tests.yml — weekly cron (0 0 * * MON) + workflow_dispatch. Runs the script with --markdown and rewrites the body of tracking issue tracking: ignored-test inventory (driven by #839) #880 via gh issue edit. Permissions: issues: write.
  • CLAUDE.md — new "Ignored-test policy (process: ignored-test policy + bitrot audit (#826 followup) #839)" section (placed after the existing TDD block). Every #[ignore] MUST have a reason string referencing either a tracking issue (#NNN) or a hardware/toolchain requirement.

First-seen-date tracking is intentionally deferred to a follow-up — this PR ships current-state inventory.

Inventory at time of PR

Run against main: 39 findings, of which 20 lack a reason string (policy violations under the new policy).

Test plan

  • uv run --no-project python ci/audit_ignored_tests.py --csv | head produces a well-formed CSV with header file,test_name,reason,line.
  • uv run --no-project python ci/audit_ignored_tests.py --markdown produces a GitHub-flavored table with summary counts.
  • Regex correctly handles #[ignore] followed by an inline // comment (e.g. crash_decoder.rs:743).
  • Skip list excludes vendored sources (.cargo/, target/).
  • Cron will fire and update issue tracking: ignored-test inventory (driven by #839) #880 on the first Monday after merge; manual gh workflow run audit-ignored-tests.yml available for early validation.

Tracking issue: #880

Closes #839

🤖 Generated with Claude Code

)

Adds visibility for #[ignore]-gated tests:

- `ci/audit_ignored_tests.py` walks the workspace and emits CSV (default)
  or markdown for every #[ignore] / #[ignore = "..."] attribute. Stdlib
  only; designed to be run via `uv run --no-project python ...`.
- `.github/workflows/audit-ignored-tests.yml` runs weekly (Mondays
  00:00 UTC) plus on workflow_dispatch, rewriting tracking issue #880
  with the current inventory in markdown.
- CLAUDE.md gains an "Ignored-test policy" section (after the TDD
  block) requiring a reason string on every #[ignore] that cites either
  a tracking issue (#NNN) or a hardware/toolchain requirement.

First-seen dates are intentionally deferred — this PR captures the
current-state inventory; baseline diffing is a follow-up.

Local smoke test (workspace `main`): script reports 39 findings, 20 of
which lack a reason string (policy violation per the new policy).

Tracking issue: #880

Closes #839

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 30, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@zackees, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 21 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b1f3aebc-99c0-4302-abe9-09f4de994199

📥 Commits

Reviewing files that changed from the base of the PR and between d17655c and bb35a51.

📒 Files selected for processing (3)
  • .github/workflows/audit-ignored-tests.yml
  • CLAUDE.md
  • ci/audit_ignored_tests.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/839-ignored-test-audit-script-cron-policy

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@zackees
zackees merged commit 4cac7e1 into main Jun 30, 2026
6 of 14 checks passed
@zackees
zackees deleted the fix/839-ignored-test-audit-script-cron-policy branch June 30, 2026 17:35
@fastled-project-sync fastled-project-sync Bot moved this to Triage in FastLED Tracker Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Triage

Development

Successfully merging this pull request may close these issues.

process: ignored-test policy + bitrot audit (#826 followup)

1 participant