feat(test): report the findings a rule's fixtures produced

[thecodedrift]

test --json reported a boolean per rule and nothing else:

{"ok":true,"rules":[{"engine":"vale","ruleId":"no-hedging","ok":true,"errors":[],"violations":[],"ran":true}]}

Every rule result now carries a findings array: the CheckResult shape check --json already prints — source, ruleId, severity, message, file, range, matchedText, and the optional note and fix — plus a bucket of "pass" or "fail" naming the fixture that produced it.

The rendered message is the point. A substitution rule whose message interpolates its captures can have the %s slots in the wrong order, fire on every fail/ fixture, stay quiet on every pass/ one, and be reported as a rule that passed. The rendered message is the only evidence otherwise, and until now the only way to see it was a second check run against a fixture path you had to construct yourself.

What changed

• schemas/verify-test.ts — findings on ruleResultSchema, defaulted to [] so the key is present in both commands' output.
• rules/vale/verify.ts — the CheckResult[] is kept beside firedIn instead of being reduced to a Set<string> of file paths. Findings are bucketed by membership in pass/.
• rules/runtime/run-fixtures.ts — execution.findings is collected per case rather than only its .length. A case counted in checkFailures contributes nothing: a harness crash arrives as one synthesized error-severity finding, and reporting that as something the rule found is the confusion checkFailures exists to prevent.
• rules/inspect.ts — findings on RuleTestResult, populated per engine and [] on every path that returns before fixtures run.
• commands/verify.ts — the human render.
• rules/fixtures.ts — FixtureFinding and FixtureBucket, shared. runtime/fixtures.ts's duplicate FixtureBucket is removed in favour of it; reference.ts's is left alone, being the corpus's own vocabulary.

Nothing new is executed, no subprocess is added, and no flag was added. The findings were already in hand and were being discarded at a seam.

Decisions worth flagging to a reviewer

• The array is always present and empty rather than absent — for a rule that produced nothing, one whose verification failed first, a refused runtime rule, verify (which runs no fixtures at all), and ast-grep. A key that is sometimes absent is one a reader learns to treat as optional, and the reading that follows is that absence means zero. There is a test per case.
• Fail-bucket findings are reported on a green --json run. They are the evidence, and they are only ever produced by a run that passed.
• The human path keeps one line per passing rule. Under a failing rule the bearing findings print beneath it, labelled by bucket, through util/format.ts's existing formatText — check's own renderer, over the CheckResult[] that is already its parameter type. Pass-bucket findings print there: pass fixture wrongly fired: <file> said that it happened and never what matched.
• No --verbose and no --include-fixtures. --verbose has zero precedent across packages/cli/src and openspec/specs; the house pattern is a rich --json beside a terse human render. An explicit fixture path already works on check, because the .taskless/** exclusion in rules/vale/run.ts is gated on wholeProject. A flag an agent has to know to pass is the ergonomics problem being filed, not its fix.
• Vale's missingFailures/unexpectedFindings are untouched. They are internal to ValeRuleVerification and decide passed; whether findings subsumes them is a separate call.
• violations[] is untouched, deliberately. Those are constraint violations — what a rule broke about its own shape. A fixture finding is what a rule reported about a document. Conflating the two is the confusion this change is correcting, so ### Requirement: A rejection names the constraint it violated is not modified.

ast-grep is out of scope, as its own future change

runTests spawns sg test and regex-parses test result: ok. N passed; N failed;. The vendored 0.45.3 binary has no --json and no output-format flag, and its fixtures are inline YAML scalars rather than files, so there is no document to attribute a finding to. Surfacing them needs a different mechanism than reading what the engine already handed us, which is the whole of what this does for the other two engines. An ast-grep rule reports findings: [] here, which is true rather than misleading — and the spec delta makes that explicitly conformant, requiring an empty array "for an engine that does not yet surface its fixture findings". So nothing archived by this PR describes behaviour the shipped code does not meet.

This PR targets main and is not part of a stack. An earlier revision of this description called it the bottom of a two-PR stack; that was wrong. Spec, implementation and archive land together here, and ast-grep gets its own proposal when that work is written rather than being a later slice of this change.

OpenSpec

### Requirement: Test runs a rule's fixtures and runs verify first is restated in full under ## MODIFIED Requirements. A MODIFIED block replaces the standing requirement, so anything left out is deleted from the spec on archive with nothing reporting it and validate --strict still passing.

Five of the eight restated scenarios have nothing to do with this change — malformed-rule ordering, ast-grep per bucket, runtime-without-flag, runtime-reaches-no-network, and check-that-throws. They are carried verbatim purely so they survive the archive. Please do not tidy them out.

Verified the way CLAUDE.md prescribes rather than trusted: first as a dry run against a throwaway SHA, then for real. cli-rule-validation went 37 → 40 scenarios and 7 → 7 requirements, with zero prior scenarios lost and exactly the three new ones added — compared by scenario-title set rather than by count, so an addition cannot mask a drop. The real archive is identical to the dry run.

The change is archived in this PR, since it targets main and the tip archives.

Consumers of @taskless/cli/schemas

zod's .parse() strips keys the schema does not declare, so a consumer still on the previously published verifyTestOutputSchema will silently drop findings from the object it hands back, on a CLI that is emitting it. Not a break — the package is pre-1.0 and this is additive, so the changeset is patch — but it is the kind of thing that generates a bug report, and the changeset says so.

Verification

• pnpm build, pnpm typecheck, pnpm lint — all clean.
• pnpm test — 103 files, 1712 tests, all passing.
• The message-order regression test was mutation-checked: rewriting the rule's message to "Replace '%s' with '%s'" (the plausible wrong order) turns three assertions red, including the human-path one. It fails when the slots are wrong.
• One existing assertion in vale-verify.test.ts was an exhaustive toEqual on the whole verification object and needed the new field. It was extended, not relaxed: both findings are spelled out with their rendered messages and ranges, so the object stays exhaustively asserted.

Refs #386

[thecodedrift]

@claude /review

[claude]

Claude finished @thecodedrift's task in 4m 57s —— View job

---

Review complete

• Read .prior-review.json for prior review context
• Gather PR diff and metadata (gh pr diff, gh pr view)
• Read changed files for full context
• Run correctness/cleanup/altitude/conventions review angles
• Verify candidate findings
• Post inline comments for confirmed findings
• Post final top-level summary

Review mode: incremental — read 0 prior review thread(s) before reviewing. .prior-review.json contains no reviewThreads, reviews, or top-level comments beyond the @claude /review trigger itself, so this is the first review of this PR — the whole diff was assessed from scratch, nothing to carry forward as addressed/still-open.

What I checked

I read the full diff (gh pr diff 396) plus the surrounding, unchanged parts of every touched function — commands/verify.ts, rules/fixtures.ts, rules/inspect.ts, rules/runtime/run-fixtures.ts, rules/runtime/fixtures.ts, rules/vale/verify.ts, schemas/verify-test.ts, util/format.ts, types/check.ts, and rules/runtime/harness.ts — tracing the findings/FixtureFinding/FixtureBucket plumbing end to end from each engine through testOneRule/verifyOneRule into the JSON schema and the human renderer.

Specifically verified, given this repo's own track record of subtle failures in exactly these spots:

• Bucket attribution in vale/verify.ts — passPaths.has(result.file) ? "pass" : "fail" reuses the exact same path-comparison convention (toRelativePosix vs. Vale's own reported file) that the pre-existing firedIn/missingFailures/unexpectedFindings logic already relied on, so this isn't a new assumption, it's the same one carried forward correctly.
• checkFailures/neverInvoked exclusion in run-fixtures.ts — both paths continue before the findings.push loop, so a harness crash or an un-invoked case correctly never contributes a synthesized finding, matching the comment's stated intent.
• findings: [] on every early-return branch in inspect.ts (sg verify-fail, vale verify-fail, vale-outcome-not-ok, runtime-refused, runtime-not-discovered, runtime-fixture-read-error) — all six covered, no path returns a RuleTestResult missing the key.
• FixtureBucket de-duplication — confirmed nothing else in src/ or test/ imports FixtureBucket from rules/runtime/fixtures.ts (the removed local copy); the only other import site is readRuntimeFixtures, unaffected. rules/reference.ts keeps its own separate FixtureBucket, as the PR description says, and that's untouched.
• The OpenSpec MODIFIED block — this repo's CLAUDE.md calls out that a MODIFIED Requirements block replaces the standing requirement wholesale, so I diffed the new delta's prose and all 8 carried-over scenarios (titles, WHEN/THEN/AND text) against the current openspec/specs/cli-rule-validation/spec.md line by line. Every paragraph and scenario is carried verbatim, in order, under the same requirement title, with exactly 3 new scenarios appended — consistent with the PR's own archive dry-run claim (37→40 scenarios, 0 lost).
• Schema/changeset — fixtureFindingSchema matches CheckResult field-for-field, .default([]) on findings correctly backstops RuleVerification results (which never set the key) at the JSON boundary. The changeset is patch, single file, on the bottom-of-stack branch, consistent with this repo's 0.y.z/stacking rules.

I did not find a correctness bug, a removed/unre-established invariant, or a CLAUDE.md violation in the diff. No inline comments posted.
(branch: feat/test-show-fixture-findings)

[thecodedrift]

Re: @claude[bot] — "Claude finished @thecodedrift's task in 4m 57s … ### Review complete…"
#396 (comment)

Read in full, and no code change was made, because the review found nothing to change: its closing line is "I did not find a correctness bug, a removed/unre-established invariant, or a CLAUDE.md violation in the diff. No inline comments posted." It surfaced as medium only as a bucketing artifact — a top-level comment with no h:/m:/l: marker defaults there — not because it raised a finding. There is no root cause to trace and no nearby instance to sweep for; acting on it would mean inventing work the review did not ask for.

Two invariants it verified are load-bearing and deliberate, recorded here so a later pass does not "tidy" them:

• The 8 carried-over scenarios in the cli-rule-validation MODIFIED block are not redundant. A MODIFIED Requirements block replaces the standing requirement, so five scenarios unrelated to this change are restated verbatim purely so openspec archive does not delete them. Verified by archive dry-run: 37 → 40 scenarios, 0 lost.
• Harness-crash findings are excluded from findings on purpose. A throw or timeout arrives as one synthesized error-severity finding; reporting it as something the rule found is exactly the confusion checkFailures exists to prevent. The continue before the collection loop is the mechanism, not an oversight.

CI is green (Validate, Changeset, Demo rules, OpenSpec Label — 5 passed, 0 failed), and the gates were re-run on the pushed tree: pnpm build, pnpm typecheck, pnpm lint clean, pnpm test 103 files / 1712 tests passing.

— AI Coding Agent