diff --git a/packages/cli/src/agent/create-vale-rule.md b/packages/cli/src/agent/create-vale-rule.md index 3d52e31d..671a6d03 100644 --- a/packages/cli/src/agent/create-vale-rule.md +++ b/packages/cli/src/agent/create-vale-rule.md @@ -1,4 +1,4 @@ -# Topic: create-vale-rule (CLI v%(CLI_VERSION)s / topic v11) +# Topic: create-vale-rule (CLI v%(CLI_VERSION)s / topic v12) ## You are here This is `create-vale-rule`. It helps you write a Vale rule: a check over @@ -219,6 +219,51 @@ it. three. If you want prose and inline code but not fenced blocks, write the list, `raw` is not "a bit wider", it is everything. + **Front matter is text, to `raw` and to every other scope but its + own.** A product-casing rule under `scope: raw` flagged + `target: taskless` in a blog post's front matter, where the lowercase + value is a machine key the publish path reads, and capitalizing it + would have broken the corpus. That is not a `raw` peculiarity: + measured on Vale v%(VALE_VERSION)s with the same rule, `text`, the + default scope and `[raw, code, text]` all reported the value on the + front-matter line too, so dropping `scope: raw` on its own fixes + nothing. `paragraph` and `sentence` skip it, and `frontmatter.` + matches only that key and never leaks into body prose; `~frontmatter` + also left the key alone and fired on the body, but chained onto `raw` + (`raw & ~frontmatter`) it matched nothing at all, so the negation + does not rescue a `raw` rule. Three fixes, then: `scope: paragraph` + or `sentence` when the rule is about prose, `frontmatter.` when + it is about a field, and in either case a `pass/` fixture that + carries the machine key, so the rule cannot regress onto it without + `test` saying so: + + ```markdown + --- + target: taskless + --- + Taskless compiles the correction into a rule. + ``` + + The first line is a machine key, read by the publish path. The last + is prose, correctly cased. A rule that fires on this document is + reading the wrong one. + + The same reach is useful in the other direction. `occurrence` with + `min: 1` under `scope: raw` can require a front-matter field to + exist, because `raw` reads the key and the value as one unparsed + line. Measured: this fires on a document whose front matter has a + `title` and no `description`, fires on one with no front matter at + all, and stays quiet once the field is there. + + ```yaml + extends: occurrence + message: "A blog draft needs a description in its frontmatter." + level: error + scope: raw + min: 1 + token: '(?m)^description: .+$' + ``` + **Vale drops everything inside a `
` element.** Measured: a `
` nested in `
` is invisible to *every* scope, `text` and `raw` included, so a `scope: figure.caption` rule over a @@ -452,8 +497,49 @@ it. already equals the replacement.** Measured with `Github: GitHub`: `github` and `Github` are flagged, `GitHub` is not. You do not need `ignorecase: false` to protect the correct spelling. - - `raw` takes a full regex when `tokens` is too restrictive; - `nonword` removes the implicit boundaries. + - `raw` takes a full regex, used verbatim, when `tokens` is too + restrictive; `nonword` removes the implicit boundaries from + `tokens`. + - **`raw` entries concatenate, they do not alternate.** `tokens` is + a list of alternatives. `raw` is not: Vale joins every entry into + one pattern, back to back, with nothing between them. Measured on + Vale v%(VALE_VERSION)s with two entries, `\bstops ` and `being\b`: + `It stops being.` fires, and a document holding either half alone + does not. So a second entry added to widen a rule never fires on + its own, and its `fail/` fixture fails, which is how this was + found. Alternation goes inside one entry, as `(a|b|c)`: + + ```yaml + # wrong: the second entry is appended to the first, so the new + # branch never fires on its own + extends: existence + raw: + - "\\bstops being\\b[^.!?\\n]{0,80}\\band becomes\\b" + - "[A-Z][^.!?\\n]{3,70}[.!?] (The|That|This|It)('s| is| was) the (problem|twist|point)\\b\\." + ``` + + ```yaml + # right: one entry, the branches alternated inside it + extends: existence + raw: + - "(\\bstops being\\b[^.!?\\n]{0,80}\\band becomes\\b|[A-Z][^.!?\\n]{3,70}[.!?] (The|That|This|It)('s| is| was) the (problem|twist|point)\\b\\.)" + ``` + + No `nonword` there, and none is needed: `nonword` governs + `tokens`, which Vale wraps in `\b…\b` unless it is set, and a + `raw` entry is used verbatim. Measured, `nonword: true` beside a + `raw` list changes nothing. Write the boundaries you want into the + pattern, as above. + + `verify` reports a `raw` list with more than one entry, on the + rule's `notice` rather than in `errors`: + + ``` + : raw has N entries; Vale joins them into one pattern with no separator, so the second never matches on its own. Write one entry with (a|b) unless the join is intended. + ``` + + If the join is intended, and the pieces are only split for + readability, the notice is yours to read and set aside. - **A token made only of punctuation can never match without `nonword: true`.** The boundaries above are `\b`, which needs a word character on the inside. An em dash has none, on either side. @@ -763,6 +849,55 @@ it. Do that once, on a file you have deliberately made violate the rule, before you believe the rule works. + **A voice or tone rule is a floor, not a ceiling.** A regex catches + the shape it was written for and nothing beyond it, so a rule about + how prose sounds only ever holds the misses already measured. From a + repository running eleven such rules: an antithesis rule keyed on + "not" and "never", the sentence a reader called out had neither, and + twelve days later `The correction was right. The place it landed was + the problem.` passed the widened rule for the same reason. Treat + every miss as the next branch, and grow the fixtures with it: + + - The sentence that got through becomes a `fail/` fixture, dated. + - The legitimate uses of the same words go into `pass/` in the same + change, so the branch cannot drift wider than what was measured. + + ``` + I've seen that read as a hole in his argument. It's the frame proving itself. (fail, first miss) + The correction was right. The place it landed was the problem. (fail, twelve days on) + Two collections beats one. This is the answer we shipped in June. (pass, must keep passing) + ``` + + **Run a new branch over the repository's own prose before it ships + at `warning`, and record the count beside the branch.** Fixtures say + the branch fires where it should; only the corpus says how often it + fires where it should not, and a rule with no number gets switched + off the first week it fires on real prose. A verdict-noun branch + that read as broad returned one hit outside its fixtures. Keep the + count in a comment next to the pattern, dated, so the next author + knows what the branch cost when it was written: + + ```yaml + # Every branch measured against the repo's 1,100 markdown files on + # 20 September. Corpus false positives per branch: 0, 0, 4, 2, 1. All + # of the four were written by the assistant, never by the human author. + ``` + + There is no single command that runs one Vale rule over the project + and counts (taskless/cli#379 tracks one). `test` isolates the rule + but sees only its fixtures. The workable way is a whole-project + `check` filtered afterwards by rule id: + + ``` + %(TASKLESS_CLI)s check --json | jq '[.results[] | select(.ruleId == "")] | length' + ``` + + A finding carries no record of which alternative matched, so to + count one branch, run the rule with that branch as its only `raw` + entry, note the number, then fold it back into the alternation. Drop + `| length` to read the hits themselves, which is what tells you + whether a hit is a false positive or a real one nobody had noticed. + 6. **Verify, then test.** Two commands, both taking the rule's directory as their argument, both run from the project root: @@ -817,6 +952,66 @@ it. `fail/` while a `warning` rule exits 0 and both are correct. `test` answers pass-or-fail; `check` shows you the finding. + **An empty `results` is a clean pass only when nothing else in the + envelope says otherwise.** Two things can leave a file unchecked + while `results` reads `[]` for it, and both are reported, so read + for them before you believe the silence: + + - **A `vale-parse-error` finding.** A document whose front matter + Vale cannot parse used to abort the whole invocation, and `check` + returned `[]` for the run, indistinguishable from a clean pass. + Now the run retries around the file Vale blames and files it as a + finding of its own, at `severity: "error"`, so a file that + could not be read never reads as a file with nothing to report. + Measured, with an unquoted colon in a front-matter value: + + ```json + {"source":"vale","ruleId":"vale-parse-error","severity":"error", + "message":"Vale could not check this file: E201: yaml: mapping values are not allowed in this context", + "file":"broken.md", …} + ``` + + Every other file's findings are reported normally, and no rule of + yours ran over this one. If it is a fixture, `test` shows the same + gap from the other side: a `fail/` fixture Vale cannot parse + reports as `fail fixture did not fire`, and a `pass/` fixture it + cannot parse stays green, since a rule that never ran cannot fire. + Fix the front matter rather than the pattern. + + - **A `notices` entry.** Under `--json` the key is absent when there + is nothing to say, so its presence is the signal. One notice + reads `Vale did not check N file(s): …` and names documents in a + format this build has no converter for, which `check` excludes + rather than letting one of them take the run down; another begins + `Vale reported while running:` and carries whatever Vale wrote to + stderr on a run that still exited zero. The one to expect there is + the `W101` warning about a rule assignment placed above any + matcher, the mistake step 4 warns of. `test` surfaces the same + text as a per-rule `notice`, printed even on a pass. + + A `failures` key is the third case and the loud one: the engine was + present and did not finish (a malformed rule, a timeout), `success` + is `false`, and `results` holds only what the other engines + reported. Only `results: []` with no `vale-parse-error` finding, no + `notices` and no `failures` is the clean pass it looks like. + + **The binary that answers is the vendored one, and only that one + counts.** `verify`, `test` and `check` all run the Vale that ships + with this CLI, v%(VALE_VERSION)s, which is what every measurement in + this recipe was taken on. A bare `vale` on your `PATH` proves nothing + about what `check` will report: one first attempt validated a rule + set against Homebrew's 3.15 while Taskless pinned 3.20, and rules + that passed on one and not the other told nobody anything. Do not + run `vale` directly, and do not add config to make a bare run + behave. The matcher that comes from doing so is `[.taskless/**]` + with the rule set to `NO`, meant to keep a bare run quiet over + fixtures that hold violations on purpose. `check` excludes + `.taskless/` before Vale runs on a whole-project walk, so that block + only ever acts under the invocation this paragraph tells you not to + use, and `verify` reports it as unnecessary (step 4 lists the + advisory). A rule needs the matchers for the files it is about and + no more. + When a `fail/` document does not fire, work down this list before touching the pattern. The cause is usually further up: - Does the rule have a `.vale.ini` at all? diff --git a/packages/cli/src/agent/verify-rule.md b/packages/cli/src/agent/verify-rule.md index 659f5378..27d05f35 100644 --- a/packages/cli/src/agent/verify-rule.md +++ b/packages/cli/src/agent/verify-rule.md @@ -1,4 +1,4 @@ -# Topic: verify-rule (CLI v%(CLI_VERSION)s / topic v3) +# Topic: verify-rule (CLI v%(CLI_VERSION)s / topic v4) ## Goal Check that a rule is well-formed (`verify`), then run its tests @@ -85,6 +85,22 @@ Both commands answer in the same shape, one entry per rule: never executed, and false for a runtime rule the execution policy refused. The second case also carries `refused` with the reason, and is the one where `ok` alone would mislead you. +- `notice` is something true about the rule that is not a failure, and + it is reported on a pass as well. For a Vale rule it carries what + Vale wrote to stderr on a run that still exited zero, such as the + `W101` warning about an assignment placed above any matcher: the rule + verifies, Vale ignores it, and only this field says so. Read it + before `ok`, the same as `ran`. + +A green Vale `test` also assumes Vale read every fixture. A fixture +whose front matter Vale cannot parse is excluded from the run and +reported as a `vale-parse-error` finding under `check`, but `test` +counts only findings from the rule under test. Measured: such a +document under `fail/` reports `fail fixture did not fire`, which is +loud; the same document under `pass/` stays green even when it holds a +violation, because a rule that never ran cannot fire. When a `fail/` +fixture reports not firing and the pattern looks right, check the +fixture's front matter before the rule. Without `--json` the same information prints as a `✓`/`✗`/`○` line per rule with its errors indented beneath. `○` is a rule that did not run.