Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/vale-recipe-regex-and-fixtures.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@taskless/cli": patch
---

`agent create-vale-rule` (topic v13) corrects two claims that cost rule authors work. Vale patterns are not RE2-only: Vale compiles with Go's `regexp` and falls back to `regexp2`, so lookahead, lookbehind and backreferences work, and the recipe no longer tells you to split a rule that one pattern expresses. Two silent limits are measured and documented alongside it: a backreference does nothing as a `swap` key, and the implicit word boundary on `tokens`/`swap` lands after a trailing lookahead, so that lookahead has to peek at a non-word character. `check` on a rule's `.tests/fail` bucket is a supported way to read a rendered message, because `.taskless/` is excluded from the whole-project walk only; when that bucket comes back empty, the recipe now sends you to the rule's own config, specifically a `[.taskless/**]` matcher, before the pattern.

`verify` carried the same imprecision and now states both halves: a `[.taskless/**]` matcher is unnecessary on a whole-project check, AND it silences the rule on a path you name, such as the rule's own fixture bucket. It was previously described as acting only under a bare `vale` invocation, which read as harmless. `agent update` (topic v10) is corrected to match.
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-09-22
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
## Why

taskless/cli#370 and #371 corrected the recipe and the `verify` advisory: a
`[.taskless/**]` matcher is not harmless. `check` excludes `.taskless/` from a
_whole-project walk_ only, so the matcher does nothing there — but on a path
named explicitly, such as the rule's own `.tests/fail` bucket, the matcher is
live and silences the rule. That is the shape that reproduces "`test` says the
fixture fired, `check` on the same fixture says nothing".

The recipe, the `update` ledger and the advisory string now all say both halves.
The standing spec does not. `cli-vale-rule-engine` still reads "`check` excludes
that tree before Vale runs, so the matcher acts only under a bare `vale`
invocation", and its scenario still requires `verify` to "report that `check`
already excludes that tree" — the exact imprecise phrasing the code no longer
ships. The spec is the source of truth for this capability, so leaving it
disagreeing with the advisory it describes is how the next author reproduces
#370 from the spec instead of the recipe.

This change carries no code. The implementation already landed in this PR; the
spec is what is behind.

## What Changes

- **`cli-vale-rule-engine`** — the advisory bullet and the
`.taskless/**` scenario state both halves of the behaviour: unnecessary on a
whole-project check, AND silencing on a named path. The scenario also gains
the `check`-notices half that the other advisory scenarios already carry, so
the two advisory paths are specified alike.

No requirement is added or removed, and no behaviour changes: this is the spec
catching up to an advisory string and a recipe that already shipped.

## Capabilities

### New Capabilities

None.

### Modified Capabilities

- `cli-vale-rule-engine`: "A rule's Vale config is validated against a schema
before it is assembled" — the `.taskless/**` advisory bullet and its scenario
are restated to match the shipped advisory.

## Impact

Documentation only. No source file, test, or public surface changes. The bump
stays `patch` and rides the existing changeset for this PR; no second changeset
is added.

## Delivery shape

**Single PR.** The spec correction is two edits inside one requirement and
belongs with the code change that made the standing text wrong, which is this
PR. It is the tip, so the change is archived here.
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
## MODIFIED Requirements

### Requirement: A rule's Vale config is validated against a schema before it is assembled

The system SHALL parse each rule's `.vale.ini` into an ordered, lossless AST and validate that AST against a schema keyed by the rule's directory id, before the config is assembled into the run config and when the rule is verified. Validation SHALL be performed on the parsed structure, never by matching the file's text.

The schema SHALL reject a config that:

- assigns any property above its first matcher (`StylesPath`, `MinAlertLevel`, or anything else; Vale ignores such a line with a `W101` warning and the rule verifies clean while enabled nowhere)
- declares a matcher without a `tskl) rule = <id>` breadcrumb naming this rule
- assigns a key other than `<id>.<id>` (a `<style>.<check>` key naming any other rule is a cross-rule override)
- assigns a value other than `YES` or `NO`
- assigns `BasedOnStyles`, with any value, empty included (on Vale 3.22.0 an empty value clears every earlier matcher's settings for the file, which in the assembled config silences every other rule whose glob reaches it; a named style loads alongside every overlapping rule; and no rule config ever needed either, since no bundled style loads unless a run-level `BasedOnStyles` names one)
- declares no matcher
- never assigns `<id>.<id> = YES` in its final per-matcher verdicts (matchers with the same glob are folded, as Vale merges them, and the last assignment wins, so a `YES` that a later `NO` in the same matcher overrides does not count)
- declares a `NO`-verdict matcher before every `YES`-verdict matcher (no style is loaded, so a rule is off until a `YES`, and such a `NO` is either dead or overridden by the `YES` that follows; no config means it)

The schema SHALL report, without rejecting, a config that:

- assigns the same key twice inside one matcher (Vale 3.21.0 keeps the last assignment; 3.20.0 kept the first)
- declares a `[*]` matcher
- declares a matcher under `.taskless/**` (`check` excludes that tree from a whole-project walk, so the matcher is unnecessary there; it is not harmless, because it silences the rule on a path named explicitly, such as the rule's own fixture bucket under `.taskless/`)

A rejected config SHALL refuse the Vale engine for that `check` run: the engine reports a failure naming the rule and the offending line, that failure SHALL reach the exit code, and other engines SHALL still run. A rule with a rejected config SHALL NOT be silently omitted from the assembled config, because a rule that is present, verifies, and reports nothing is the silent-disable failure this engine's design exists to prevent. Advisories SHALL be surfaced as notices and SHALL NOT affect the exit code.

Assembly SHALL write each accepted config's source verbatim. The parsed structure is read for validation and for the list of matcher patterns; it is not re-serialized.

#### Scenario: A foreign assignment refuses the run

- **WHEN** `no-simply/.vale.ini` assigns `no-hedging.no-hedging = NO` and `check` runs
- **THEN** the Vale engine SHALL report a failure naming `no-simply` and that line
- **AND** the exit code SHALL be non-zero
- **AND** ast-grep results for the same run SHALL still be reported

#### Scenario: A run-level key in a rule config is rejected

- **WHEN** a rule's config places `StylesPath = .` above its first matcher
- **THEN** `verify` SHALL reject the rule, naming the key and the line
- **AND** `check` SHALL refuse the Vale engine rather than strip the line

#### Scenario: A matcher without a breadcrumb is rejected

- **WHEN** a rule's config declares `[*.md]` with `<id>.<id> = YES` and no `tskl) rule` key
- **THEN** `verify` SHALL reject the rule, naming the matcher

#### Scenario: A disable that precedes every enable is rejected

- **WHEN** a rule's config declares `[docs/legacy/**]` with `<id>.<id> = NO` and then `[docs/**]` with `<id>.<id> = YES`
- **THEN** `verify` SHALL reject the rule, naming the `NO` matcher and the `YES` that re-enables it

#### Scenario: A BasedOnStyles assignment is rejected

- **WHEN** a rule's config sets `BasedOnStyles =` inside a matcher
- **THEN** `verify` SHALL reject the rule under `vale-config-no-based-on-styles`, naming the line and saying that the line silences the other rules whose globs overlap
- **AND** a `BasedOnStyles` naming a style SHALL be rejected under the same constraint

#### Scenario: A [formats] section is rejected as a matcher it cannot be

- **WHEN** a rule's config declares a `[formats]` section
- **THEN** `verify` SHALL reject it for the breadcrumb it lacks and the foreign key it assigns, so no rule config can move a file between parser tiers

#### Scenario: A repeated key is reported, not rejected

- **WHEN** one matcher assigns `<id>.<id> = YES` and then `<id>.<id> = NO`, and an earlier matcher assigns `<id>.<id> = YES`
- **THEN** `verify` SHALL accept the rule and report the repeat as a notice
- **AND** `check` SHALL run the Vale engine and carry the same text in its notices

#### Scenario: A rule whose only enable is overridden is rejected

- **WHEN** the only matcher assigning `<id>.<id> = YES` later assigns `<id>.<id> = NO`, in the same section or in a second section with the same glob
- **THEN** `verify` SHALL reject the rule as present but off, under `vale-config-enabled-somewhere`
- **AND** the repeat SHALL still be reported as a notice

#### Scenario: A `.taskless/**` matcher is reported as unnecessary

- **WHEN** a rule's config declares a matcher under `.taskless/**`
- **THEN** `verify` SHALL accept the rule and report both halves: that the matcher is unnecessary on a whole-project check, which excludes `.taskless/` before Vale runs, AND that it silences the rule on a path named explicitly, such as the rule's own fixture bucket
- **AND** `check` SHALL carry the same text in its notices

#### Scenario: An accepted config is assembled byte-for-byte

- **WHEN** a config passes the schema
- **THEN** the assembled run config SHALL contain that file's bytes unchanged under the rule's breadcrumb comment
- **AND** the matcher patterns reported for the run SHALL equal the section names in the parsed structure
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
## 1. Spec

- [x] 1.1 Restate the `.taskless/**` advisory bullet in
`cli-vale-rule-engine` to say it is unnecessary on a whole-project check
AND silencing on a named path.
- [x] 1.2 Restate the `A .taskless/** matcher is reported as unnecessary`
scenario to require both halves, plus the `check`-notices half the other
advisory scenarios carry.
- [x] 1.3 Confirm the MODIFIED block restates the requirement in full: same
title, all ten scenarios, and every bullet of both lists.
- [x] 1.4 Dry-run `openspec archive` and diff the scenario count before and
after to prove nothing is dropped.
5 changes: 3 additions & 2 deletions openspec/specs/cli-vale-rule-engine/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ The schema SHALL report, without rejecting, a config that:

- assigns the same key twice inside one matcher (Vale 3.21.0 keeps the last assignment; 3.20.0 kept the first)
- declares a `[*]` matcher
- declares a matcher under `.taskless/**` (`check` excludes that tree before Vale runs, so the matcher acts only under a bare `vale` invocation)
- declares a matcher under `.taskless/**` (`check` excludes that tree from a whole-project walk, so the matcher is unnecessary there; it is not harmless, because it silences the rule on a path named explicitly, such as the rule's own fixture bucket under `.taskless/`)

A rejected config SHALL refuse the Vale engine for that `check` run: the engine reports a failure naming the rule and the offending line, that failure SHALL reach the exit code, and other engines SHALL still run. A rule with a rejected config SHALL NOT be silently omitted from the assembled config, because a rule that is present, verifies, and reports nothing is the silent-disable failure this engine's design exists to prevent. Advisories SHALL be surfaced as notices and SHALL NOT affect the exit code.

Expand Down Expand Up @@ -288,7 +288,8 @@ Assembly SHALL write each accepted config's source verbatim. The parsed structur
#### Scenario: A `.taskless/**` matcher is reported as unnecessary

- **WHEN** a rule's config declares a matcher under `.taskless/**`
- **THEN** `verify` SHALL accept the rule and report that `check` already excludes that tree
- **THEN** `verify` SHALL accept the rule and report both halves: that the matcher is unnecessary on a whole-project check, which excludes `.taskless/` before Vale runs, AND that it silences the rule on a path named explicitly, such as the rule's own fixture bucket
- **AND** `check` SHALL carry the same text in its notices

#### Scenario: An accepted config is assembled byte-for-byte

Expand Down
85 changes: 70 additions & 15 deletions packages/cli/src/agent/create-vale-rule.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Topic: create-vale-rule (CLI v%(CLI_VERSION)s / topic v12)
# Topic: create-vale-rule (CLI v%(CLI_VERSION)s / topic v13)

## You are here
This is `create-vale-rule`. It helps you write a Vale rule: a check over
Expand Down Expand Up @@ -471,15 +471,42 @@ it.
expect to add to the list.

3. **Know what you are writing: `tokens` and `swap` keys are patterns,
not literals.** They compile as **Go RE2** regular expressions.
not literals.** They compile as Go regular expressions.

*This step is about `tokens` and `swap` only. A `capitalization`,
`occurrence` or `metric` rule has neither, skip to step 4.*

- `(?:…)`, `[…]`, `|`, `+`, `?` all work.
- **Lookahead and lookbehind do not exist in RE2.** A rule that needs
"X but not when followed by Y" cannot be written as a single
`substitution`; split it or narrow with `scope`.
- **Lookaround and backreferences work, and they are not free.**
Vale compiles a pattern with Go's own `regexp` first and falls
back to `regexp2` when that engine refuses it, so `(?=…)`,
`(?<=…)` and `\1` are all available even though Go's `regexp` has
none of them. Measured on Vale v%(VALE_VERSION)s with throwaway
rules, each firing on its `fail/` fixture and quiet on `pass/`: a
repeated-word `\b(\w+) \1\b`, a `foo(?= bar)` lookahead, and a
`(?<=x )y` lookbehind. The fallback engine backtracks and is the
Comment thread
thecodedrift marked this conversation as resolved.
slower of the two, so keep lookaround off a pattern that runs
over every file in the project, and prove any rule that uses one
with a fixture rather than trusting the syntax. "X but not when
followed by Y" is therefore writable as a single `substitution`,
but splitting it or narrowing with `scope` is still the cheaper
rule when either will do.

Two measured limits sit on top of that, and both are silent:

- **A backreference does nothing in `swap`.** The same
`(\w+) \1` that fires under `tokens` and under `raw` produces
no finding as a `swap` key, with nothing on stderr and the
rule loading cleanly, so the rule looks healthy and never
fires at all. A repeated-word check has to be an `existence`
rule; it cannot be a `substitution`.
- **A trailing lookahead in `tokens` or `swap` has to peek at a
non-word character.** The implicit `\b` is appended after the
lookahead (the lookahead is zero-width, so the position is
still where the match ended), which puts the boundary between
the match and the text peeked at. `foo(?= bar)` fires;
`foo(?=bar)` can never match, whatever the document says. Use
`raw` when the lookahead has to land on a word character.
- **Word boundaries are applied for you, around the whole pattern.**
Measured: `Github` does not fire inside `GithubToken`, and the
multi-word `click here` does not fire inside `Clicking here`.
Expand Down Expand Up @@ -701,8 +728,9 @@ it.
the rule: if it was the only `YES`, the config is rejected, not
advised
- a `[*]` matcher, which reaches every file Vale can read
- a matcher under `.taskless/**`, which `check` already excludes
before Vale runs, so it acts only under a bare `vale` invocation
- a matcher under `.taskless/**`, which a whole-project `check`
already excludes, and which silences the rule over a fixture
bucket you name on purpose (step 6)

Each advisory has a legitimate reading, which is what separates the
two lists. Fix a rejection before moving on; read an advisory and
Expand Down Expand Up @@ -939,13 +967,35 @@ it.
reported one entry per rule. `.taskless/rules/vale` covers every Vale
rule; no argument at all covers the project.

If you would rather see the raw findings, the message text and the
line numbers, run `check` against a bucket instead:
**`test` answers pass-or-fail and never shows you the finding**, so
it cannot tell you that a `substitution` message renders its two
`%%s` slots in the wrong order: the rule fires, the fixture is
satisfied, and `test` reports `ok`. To read the rendered message,
the line numbers and the matched text, name the bucket to `check`:

```
%(TASKLESS_CLI)s check .taskless/rules/vale/<id>/.tests/fail --json
```

**That works because a path you name is honored.** `.taskless/` is
excluded from the *whole-project* walk only, so a bare `check` over
the project reports nothing from anyone's fixtures while the command
above reports every finding in that bucket. Measured on this build:
a whole-project `check` returned no result under `.taskless/`, and
the same rule's `fail/` bucket named explicitly returned its
findings with the message text rendered.

**If that command returns `results: []` for a rule whose `test` is
green, suspect the rule's own config before the pattern.** A
`[.taskless/**]` matcher setting `<id>.<id> = NO` turns the bucket
off for exactly this invocation, which is the one shape that
reproduces "`test` says the fixture fired, `check` on the same
fixture says nothing". Measured: adding that matcher to a working
rule left `test` at `ok: true` and emptied `results`. The tell is
in the same envelope, as a notice reading `matcher [.taskless/**]
is unnecessary`. `verify` reports it too, as an advisory. Delete
the matcher; step 4 explains why no rule needs one.

Read `results` there. Ignore `success` and the exit code: `success`
says the run worked rather than that the fixture behaved, and the
exit code follows severity, so a `level: error` rule exits 1 on
Expand Down Expand Up @@ -1005,15 +1055,20 @@ it.
run `vale` directly, and do not add config to make a bare run
behave. The matcher that comes from doing so is `[.taskless/**]`
with the rule set to `NO`, meant to keep a bare run quiet over
fixtures that hold violations on purpose. `check` excludes
`.taskless/` before Vale runs on a whole-project walk, so that block
only ever acts under the invocation this paragraph tells you not to
use, and `verify` reports it as unnecessary (step 4 lists the
advisory). A rule needs the matchers for the files it is about and
fixtures that hold violations on purpose. It does not stay confined
to the invocation it was written for: a whole-project `check` skips
`.taskless/` without its help, and on a fixture bucket you name it
is the one thing acting, which is how it empties the `check` above
while `test` stays green. `verify` reports it as an advisory (step 4
lists it). A rule needs the matchers for the files it is about and
no more.

When a `fail/` document does not fire, work down this list before
touching the pattern. The cause is usually further up:
- Read the finding first, with the `check` on the `fail/` bucket
above. What the run saw is cheaper than any guess about why it
saw nothing, and it separates "no finding" from "a finding whose
message is wrong".
- Does the rule have a `.vale.ini` at all?
- Is the assignment underneath a `[…]` matcher?
- Is it spelled `<id>.<id>`, both halves the same?
Expand Down Expand Up @@ -1191,7 +1246,7 @@ either:

**The id must be word characters only.** `consistency` is the one
extension point that compiles the rule's own name into the pattern, as
a `(?P<id>…)` capture group, and Go RE2 rejects a group name containing
a `(?P<id>…)` capture group, and Go's `regexp` rejects a group name containing
a hyphen. Measured: an id of `ize-ise` fails with `E201 … invalid group
name` and takes **every** Vale rule in the project down with it, because
Vale reads one config for the whole run. Name this one `izeise` or
Expand Down
Loading
Loading