Split out of #370, where it came up as the more useful of two options. #370 itself is fixed in the recipe; this is the CLI half, filed separately because it is a test ergonomics question and not a check scoping one.
test answers one question: did the fail fixture fire and the pass fixture stay quiet. It never shows what fired.
$ pnpm cli test .taskless/rules/vale/no-hedging --json
{"ok":true,"rules":[{"engine":"vale","ruleId":"no-hedging","ok":true,"errors":[],"violations":[],"ran":true}]}
The violations array is not the rule's findings. packages/cli/src/schemas/verify-test.ts:15 types it as constraint violations, each carrying a constraintId that is "Stable id of a constraint published in @taskless/cli/reference.json" — the authoring constraints verify enforces, not what the rule matched in the fixture.
So a rule can be green with its message wrong. That is not hypothetical: a substitution rule shipped with its two %s slots in the wrong order, test reported ok, and the swap was only caught by running check over a real corpus file and reading the rendered message. Nothing about a fixture run would have shown it, because the fixture run never renders the message.
The workaround is real and now documented in create-vale-rule (see #370): check honors an explicit path, so check .taskless/rules/vale/<id>/.tests/fail --json does return the findings with their rendered messages. It works because the .taskless/** exclusion at packages/cli/src/rules/vale/run.ts:576 applies only on a whole-project walk. But it means the author runs a second, different command to see what the first one already saw.
What to do
test --verbose (or a field on the existing --json output) printing the findings each fixture produced: rendered message, file, line, and which fixture bucket it came from.
- The rendered message is the point. A finding whose message reads correctly in the JSON is the only evidence that
%s slots are in the right order.
- Cover both engines. An ast-grep rule's message has the same failure mode.
- Decide whether a pass-fixture finding (which makes
ok false) should print by default rather than behind the flag, since that is the case where the author most needs to see what matched.
- Do NOT add
--include-fixtures to check. An explicit fixture path already works; a flag for it would reach a path that is already reachable.
Refs #370
Split out of #370, where it came up as the more useful of two options. #370 itself is fixed in the recipe; this is the CLI half, filed separately because it is a
testergonomics question and not acheckscoping one.testanswers one question: did the fail fixture fire and the pass fixture stay quiet. It never shows what fired.The
violationsarray is not the rule's findings.packages/cli/src/schemas/verify-test.ts:15types it as constraint violations, each carrying aconstraintIdthat is "Stable id of a constraint published in@taskless/cli/reference.json" — the authoring constraintsverifyenforces, not what the rule matched in the fixture.So a rule can be green with its message wrong. That is not hypothetical: a
substitutionrule shipped with its two%sslots in the wrong order,testreportedok, and the swap was only caught by runningcheckover a real corpus file and reading the rendered message. Nothing about a fixture run would have shown it, because the fixture run never renders the message.The workaround is real and now documented in
create-vale-rule(see #370):checkhonors an explicit path, socheck .taskless/rules/vale/<id>/.tests/fail --jsondoes return the findings with their rendered messages. It works because the.taskless/**exclusion atpackages/cli/src/rules/vale/run.ts:576applies only on a whole-project walk. But it means the author runs a second, different command to see what the first one already saw.What to do
test --verbose(or a field on the existing--jsonoutput) printing the findings each fixture produced: rendered message, file, line, and which fixture bucket it came from.%sslots are in the right order.okfalse) should print by default rather than behind the flag, since that is the case where the author most needs to see what matched.--include-fixturestocheck. An explicit fixture path already works; a flag for it would reach a path that is already reachable.Refs #370