Skip to content

test: show the findings a fixture produced, not just pass or fail #386

Description

@thecodedrift

Split out of #370, where it came up as the more useful of two options. #370 itself is fixed in the recipe; this is the CLI half, filed separately because it is a test ergonomics question and not a check scoping one.

test answers one question: did the fail fixture fire and the pass fixture stay quiet. It never shows what fired.

$ pnpm cli test .taskless/rules/vale/no-hedging --json
{"ok":true,"rules":[{"engine":"vale","ruleId":"no-hedging","ok":true,"errors":[],"violations":[],"ran":true}]}

The violations array is not the rule's findings. packages/cli/src/schemas/verify-test.ts:15 types it as constraint violations, each carrying a constraintId that is "Stable id of a constraint published in @taskless/cli/reference.json" — the authoring constraints verify enforces, not what the rule matched in the fixture.

So a rule can be green with its message wrong. That is not hypothetical: a substitution rule shipped with its two %s slots in the wrong order, test reported ok, and the swap was only caught by running check over a real corpus file and reading the rendered message. Nothing about a fixture run would have shown it, because the fixture run never renders the message.

The workaround is real and now documented in create-vale-rule (see #370): check honors an explicit path, so check .taskless/rules/vale/<id>/.tests/fail --json does return the findings with their rendered messages. It works because the .taskless/** exclusion at packages/cli/src/rules/vale/run.ts:576 applies only on a whole-project walk. But it means the author runs a second, different command to see what the first one already saw.

What to do

  • test --verbose (or a field on the existing --json output) printing the findings each fixture produced: rendered message, file, line, and which fixture bucket it came from.
  • The rendered message is the point. A finding whose message reads correctly in the JSON is the only evidence that %s slots are in the right order.
  • Cover both engines. An ast-grep rule's message has the same failure mode.
  • Decide whether a pass-fixture finding (which makes ok false) should print by default rather than behind the flag, since that is the case where the author most needs to see what matched.
  • Do NOT add --include-fixtures to check. An explicit fixture path already works; a flag for it would reach a path that is already reachable.

Refs #370

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CLIRelated to the taskless CLI

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions