Skip to content

process: ignored-test policy + bitrot audit (#826 followup) #839

Description

@zackees

Background

The #826 audit counted ~50+ #[ignore] annotations across the test suite. Most are legitimate (require attached hardware: a real ESP32, a Teensy, a USB CDC device) but some are stale — the underlying reason for the ignore was fixed and nobody removed the annotation. Without a tracking discipline, ignored tests rot until they no longer even compile.

Examples from #826

  • crates/fbuild-build/tests/teensy30_acceptance.rs#[ignore = "requires teensy hardware"] (legitimate; hardware required)
  • crates/fbuild-build/tests/avr_build.rs#[ignore] (reason unclear; needs investigation)
  • crates/fbuild-packages/tests/disk_cache_schema_migration.rs#[ignore] (reason unclear)

(Full list to be enumerated by the script proposed below.)

Why this is NOT a dylint

This is a process concern, not a code-shape concern. The right tool is a periodic audit, not a per-commit lint.

Proposal

  1. Add ci/audit_ignored_tests.py that walks the workspace, finds every #[ignore] (with and without a reason string), and emits a CSV: file,test_name,reason,line.
  2. Add a weekly GitHub Action (cron 0 0 * * MON) that runs the script and posts the diff vs. the previous week to a tracking issue (open one stable issue tracking: ignored-test inventory and update its body).
  3. Define the policy:
    • Every #[ignore] must have a reason string (#[ignore = \"...\"]).
    • The reason must reference either a tracking issue (#NNN) OR a hardware requirement (e.g. requires teensy 3.0).
    • A test that's been ignored for 90+ days without a hardware reason gets escalated to a CodeRabbit review.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions