Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .changeset/cli-v2-rule-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ The CLI now speaks the Taskless v2 rule API, which addresses rules by their own

What you may need to react to:

- **`taskless check` fails when an issued ast-grep or Vale rule has been edited.** Logged in, every file of every rule is checked against what Taskless issued, and an edited rule of any engine does not run. For ast-grep and Vale rules the run also fails, naming each changed, removed, or added file. To fix it, run `taskless rule restore <ruleId>` rather than editing the rule back by hand. Logged out, with `--anonymous`, or with `--dangerously-run-scripts`, nothing is checked, as before. Rules you wrote yourself are unaffected.
- **`taskless check` fails when an issued ast-grep or Vale rule has been edited.** Logged in, every file of every rule is checked against what Taskless issued, and an edited rule of any engine does not run. For ast-grep and Vale rules the run also fails, naming each changed, removed, or added file. To fix it, run `taskless rule restore <ruleId>` rather than editing the rule back by hand. The same applies to an issued rule copied or renamed to a new id: the copy does not run, and the run fails naming the rule it came from. Logged out, with `--anonymous`, or with `--dangerously-run-scripts`, nothing is checked, as before. Rules you wrote yourself are unaffected.
- **`taskless check` never rewrites your rules.** It used to try to repair a changed runtime rule in the middle of a run. It now reports the change and names the command to run.
- **`taskless rule create --json` prints `requestId` instead of `ruleId`.** The old field always held the request id, never a rule id. The ids of the rules that were written are in `rules`, and those are what `taskless rule improve` takes.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-09-29
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
## Context

`applyVerdicts` (`packages/cli/src/rules/verdicts.ts`) is the pure policy that
turns a v2 reconcile answer into what `check` does with each reported rule. An
`unknown` static rule runs silently; an `unknown` runtime rule is not executed;
a `missing` rule warns. The service contract (taskless/taskless#262 design D7,
published by #264) adds `copyOf` to `unknown` entries:

```
unknown: [{ ruleId, copyOf?: { ruleId, revisionId, files: [{ path, expected?, got? }] } }]
```

`files` pairs by digest before path, so a file carried unchanged under a new
name is not listed.

## Goals / Non-Goals

**Goals:** an sg/Vale copy fails `check` naming its source and the differing
files; a copy whose source is `missing` is reported once, as a rename; nothing
changes for a response without `copyOf`.

**Non-Goals:** detecting copies locally (the CLI has no issued digests); any
change to `missing` on its own; an override flag (the contract has none).

## Decisions

### D1. Read `copyOf` defensively; do not touch the vendored schema

`applyVerdicts` already takes the response as `unknown` and parses every field
it uses, so the generated types add nothing to the policy. The vendored
`api-v2.schema.json` is documented as the live API's own account of itself;
hand-extending it with a field that is not deployed would make it claim what the
service does not yet serve, and would be overwritten by the next
`generate:api` anyway. It is re-vendored once #264 is live.

### D2. A copy stays `unknown` in the disposition and in `integrity`

The service keeps the copy in `unknown` (its D1), so the CLI does too, and adds
`copyOf` beside it. No new `IntegrityVerdict` value: a consumer keyed on the
verdict set keeps working, and `copyOf` is the additive signal. The diff goes in
the existing `files` field, which already means "what differs from what was
issued".

### D3. Rename detection: the source is in this answer's `missing` set

A copy whose `copyOf.ruleId` is answered `missing` (and was not itself reported)
is a rename. The copy's message says the source was deleted and names
`rule restore <source>`, and the source's own `missing` notice is suppressed.
The source's `missing` entry stays in `integrity`: both facts are true, and that
entry carries the `revisionId` restore brings back. The "one finding"
obligation is about what a person reads; `integrity` is per-rule state, and the
copy's `copyOf.sourceMissing: true` links the two for a machine reader.

### D4. Runtime copies: unchanged outcome, source named

A runtime `unknown` rule is never executed without `--dangerously-run-scripts`,
so the outcome stays. Its skip reason names the source, since that is the first
thing a reader will want and costs nothing. A plain runtime copy gets no notice
(its skip reason covers it, as for any runtime `unknown`). A runtime rename is
one notice, because it replaces the `missing` warning that would otherwise have
been printed; it does not fail the run, matching `unsafe` runtime rules.

### D5. A malformed `copyOf` fails closed

`copyOf` absent or `null` is a local rule. Present but not an object with a
non-empty string `ruleId` is treated as unaccounted: not run, and the run fails.
The service sends `copyOf` only when it found issued content, so even an
unreadable one says "this is a copy"; ignoring it would run exactly the rule the
field exists to stop. This follows the policy's existing rule that an answer the
CLI cannot read never makes a run greener. `revisionId` and `files` are read
tolerantly: only the source id is needed to act.

## Risks / Trade-offs

- **A local rule legitimately started from an issued rule's file fails** until
that file is changed. Intended by the service contract; the message names the
source, and the recipe says to write the rule from scratch.
- **Before #264 deploys** no response carries `copyOf`, so the behavior is
inert, not wrong.
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
## Why

Renaming a rule's directory gets around tamper detection for ast-grep and Vale
rules (taskless/taskless#255). Copy `.taskless/rules/vale/foo-1/` to `bar-2/`,
loosen its `.vale.ini`, and delete `foo-1/`. v2 reconcile answers `bar-2` as
`unknown`, which the CLI runs silently as a locally written rule, and `foo-1` as
`missing`, which only warns. `check` passes with a tampered Taskless rule
running.

The product decision on #255 keeps deletion legitimate (`missing` stays a
warning, there is no retirement) and closes the hole on the copy. The service
change (taskless/taskless#262, published in `__schema` by taskless/taskless#264)
annotates such an `unknown` rule with `copyOf: { ruleId, revisionId, files }`
when any file it reported has the content of a signed file issued in this
repository, at any path. Its `cli` spec delta states the client obligation this
change implements. 0.12.0 is the first v2 client and is unreleased, so it ships
with the obligation from the start.

## What Changes

- An sg or Vale `unknown` rule carrying `copyOf` does not run: it is removed
from the snapshot, and `check` fails naming the source rule and each file that
differs from it.
- When the source is also answered `missing`, the copy and the deletion are
reported as one rename, whose message names `taskless rule restore <source>`.
The source gets no separate `missing` warning.
- A runtime `unknown` rule with `copyOf` is not executed, exactly as before; its
skip reason now names the source. A runtime rename is one notice in place of
the `missing` warning, and does not fail the run.
- A `copyOf` that is present but unreadable fails closed: the rule is
unaccounted, does not run, and fails the run.
- `check --json`'s `integrity` gains an optional `copyOf` on `unknown` entries
(`{ ruleId, revisionId?, sourceMissing }`), with the diff in the existing
`files`. `engine.log` records each copy and whether it is a rename.
- The `check` and `recover-rule` agent recipes describe the failure and the fix
(restore the source, delete the copy).
- Unchanged: `missing` without a copy, `unknown` without `copyOf`, every `run`,
`unsafe`, and withheld outcome, accounting, and the request.

## Capabilities

### New Capabilities

_None._

### Modified Capabilities

- `cli-rule-reconciliation`: the per-engine verdict policy gains the `copyOf`
row and the rename rule; `missing` stays a warning.
- `cli-check`: the exit code fails on a static copy, and `integrity` carries
`copyOf`.

## Impact

- `packages/cli/src/rules/verdicts.ts` (policy), `plan-check.ts` (log line),
`src/schemas/check.ts` (`--json` shape).
- Tests: `test/verdicts.test.ts` rows, one end-to-end case in
`test/runtime-check.test.ts`.
- Recipes: `src/agent/check.md`, `src/agent/recover-rule.md`. Both are already
new or bumped in this unreleased cycle, so their topic versions stay.
- The vendored v2 schema (`src/generated/api-v2.*`) is NOT changed: `copyOf` is
read defensively from the untyped response, and the vendored files are
re-generated once #264 is live.
- Changeset: one sentence added to the pending `.changeset/cli-v2-rule-api.md`.

## Delivery shape

**Single PR.** The change is one branch in a pure policy function plus its
reporting, tests, recipes, and spec, well under the review budget. It is safe on
its own: without `copyOf` in a response nothing changes, so it can land before
or after the service deploys #264.
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
## MODIFIED Requirements

### Requirement: Check subcommand exit codes reflect error severity

The CLI SHALL exit with code 0 when no error-severity matches are found (including when only warnings, info, or hints exist) and no reconcile outcome below requires failure. The CLI SHALL exit with code 1 when at least one error-severity match is found. The CLI SHALL also exit with code 1, whatever the findings and in both human and `--json` modes, when a completed reconcile:

- returned a non-empty `entitlement.withheld`;
- returned an `unsafe` verdict for an `sg` or `vale` rule;
- returned an `sg` or `vale` rule in `unknown` carrying `copyOf`, or a rule of any engine carrying a `copyOf` the CLI cannot read; or
- left a reported rule unaccounted for (in none, or more than one, of `rules`, `unknown`, and `entitlement.withheld`).

On an authenticated run that would reconcile, the CLI SHALL also exit with code 1 when two rule directories under different engines share an id. A logged-out run verifies nothing and does not fail on it. Under `--json`, `success` SHALL be `false` whenever the exit code is non-zero.

#### Scenario: Exit 0 when clean

- **WHEN** the scanner produces zero results
- **THEN** the process SHALL exit with code 0

#### Scenario: Exit 0 when only warnings

- **WHEN** the scanner produces results but none have severity "error"
- **THEN** the process SHALL exit with code 0

#### Scenario: Exit 1 when errors found

- **WHEN** the scanner produces at least one result with severity "error"
- **THEN** the process SHALL exit with code 1

#### Scenario: Exit 1 when a runtime rule is withheld for entitlement

- **WHEN** reconciliation returns a non-empty `entitlement.withheld` and the scan produces zero results
- **THEN** the process SHALL exit with code 1
- **AND** under `--json`, `success` SHALL be `false`

#### Scenario: Entitlement without a withheld file does not fail

- **WHEN** reconciliation returns `entitlement.runtimeSignatures: false` with an empty or absent `withheld`, and the scan produces zero results
- **THEN** the process SHALL exit with code 0

#### Scenario: Exit 1 when a static rule was edited

- **WHEN** reconciliation returns an `unsafe` verdict for an `sg` or `vale` rule and the scan produces zero results
- **THEN** the process SHALL exit with code 1

#### Scenario: Exit 1 when a static rule is a copy of an issued rule

- **WHEN** reconciliation returns an `sg` or `vale` rule in `unknown` with `copyOf`, and the scan produces zero results
- **THEN** the process SHALL exit with code 1

#### Scenario: Exit 1 when a reported rule is unaccounted for

- **WHEN** a reported rule appears in none of `rules`, `unknown`, and `entitlement.withheld`
- **THEN** the process SHALL exit with code 1

#### Scenario: Missing does not fail

- **WHEN** reconciliation returns only `run` and `missing` verdicts and the scan produces zero results
- **THEN** the process SHALL exit with code 0

### Requirement: Check reports rule integrity under --json

Under `--json`, `taskless check` SHALL carry an additive, optional `integrity` array with one
entry per rule whose outcome is `unsafe`, `missing`, runtime `unknown`, `unknown` with
`copyOf` (any engine), `unaccounted`, or `duplicate`, each
`{ ruleId, engine?, verdict, files?, revisionId?, copyOf? }`. `files` SHALL list each
differing path with `expected` and `got` as the server returned them; for a copy it SHALL be
the server's `copyOf.files`. `copyOf` SHALL be `{ ruleId, revisionId?, sourceMissing }`, where
`sourceMissing` is whether the source rule was answered `missing`. Static `unknown` rules
without `copyOf` and `run` rules SHALL NOT appear. The field SHALL be omitted when there is
nothing to report.

#### Scenario: An edited rule appears with its differing files

- **WHEN** reconciliation returns `unsafe` for vale rule `no-simply-1a2b3c4d` with `.vale.ini` changed
- **THEN** `integrity` SHALL include `{ ruleId: "no-simply-1a2b3c4d", engine: "vale", verdict: "unsafe", files: [{ path: ".vale.ini", expected, got }] }`

#### Scenario: A clean run omits the field

- **WHEN** every reported rule is `run` or static `unknown` and nothing is `missing`
- **THEN** `check --json` SHALL NOT include `integrity`

#### Scenario: A renamed rule appears as a copy beside its missing source

- **WHEN** reconciliation returns vale rule `bar-2` in `unknown` with `copyOf: { ruleId: "foo-1", revisionId: "r1", files: [{ path: ".vale.ini", expected, got }] }` and returns `foo-1` as `missing` with `revisionId` `r2`
- **THEN** `integrity` SHALL include `{ ruleId: "bar-2", engine: "vale", verdict: "unknown", files: [{ path: ".vale.ini", expected, got }], copyOf: { ruleId: "foo-1", revisionId: "r1", sourceMissing: true } }`
- **AND** SHALL include `{ ruleId: "foo-1", engine: "vale", verdict: "missing", revisionId: "r2" }`
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
## MODIFIED Requirements

### Requirement: Reconcile verdicts are applied per engine

The CLI SHALL read the v2 reconcile response as a list of per-rule verdicts
(`rules[]`, each `{ ruleId, engine, verdict }` with `verdict` one of `run`, `unsafe`,
`missing`), a list of `unknown` rules (each `{ ruleId, copyOf? }`), and
`entitlement.withheld`, and SHALL apply this policy:

| Verdict | runtime | sg / vale |
| -------------------- | -------------------------------------------------- | --------------------------------------------- |
| `run` | execute | run |
| `withheld` | do not execute; fail `check` | (never sent) |
| `unsafe` | do not execute; name `rule restore` | do not run; fail `check`; name `rule restore` |
| `missing` | warn; name `rule restore` | warn; name `rule restore` |
| `unknown` | do not execute (needs `--dangerously-run-scripts`) | run |
| `unknown` + `copyOf` | do not execute; name the source | do not run; fail `check`; name the source |

The engine SHALL be taken from the verdict's `engine` for `rules[]` entries and from the
reporting directory for `unknown` entries. An `unsafe` notice SHALL name the rule and each
differing path, saying whether it changed, was removed, or was added. A signature SHALL
authorize running a runtime rule only through a `run` verdict, never by local comparison.

#### Scenario: An edited static rule fails and does not run

- **WHEN** reconcile returns `{ ruleId: "no-simply-1a2b3c4d", engine: "vale", verdict: "unsafe", files: [{ path: ".vale.ini", expected, got }] }`
- **THEN** the rule SHALL NOT run
- **AND** `check` SHALL exit non-zero naming the rule and `.vale.ini` as changed

#### Scenario: A locally written static rule runs

- **WHEN** reconcile lists a static rule's id in `unknown` without `copyOf`
- **THEN** that rule SHALL run
- **AND** the CLI SHALL emit no notice for it

#### Scenario: A locally written runtime rule does not execute

- **WHEN** reconcile lists a runtime rule's id in `unknown` and `--dangerously-run-scripts` is not set
- **THEN** the rule SHALL NOT execute
- **AND** its skip reason SHALL say it was not issued by the rule service

#### Scenario: Missing warns and does not fail

- **WHEN** reconcile returns a `missing` verdict for any engine, and no `unknown` rule names it in `copyOf`
- **THEN** the CLI SHALL warn naming the rule and `taskless rule restore <ruleId>`
- **AND** SHALL NOT change the exit code because of it

#### Scenario: An edited runtime rule is withheld, not failed

- **WHEN** reconcile returns an `unsafe` verdict for a runtime rule
- **THEN** the rule SHALL NOT execute
- **AND** the exit code SHALL NOT change because of that verdict alone

## ADDED Requirements

### Requirement: A copy of an issued rule does not run as a local rule

When reconcile returns an `unknown` rule carrying `copyOf` (taskless/taskless#255), the CLI
SHALL NOT run or execute it, and SHALL remove it from the snapshot the engines read. For an
`sg` or `vale` rule, `check` SHALL fail with one message naming the rule, the source rule
`copyOf.ruleId`, and each path in `copyOf.files` as changed, removed, or added. For a runtime
rule, the exit code SHALL NOT change, and its skip reason SHALL name the source and say it
was not issued by the rule service.

When `copyOf.ruleId` is also returned as `missing`, the CLI SHALL report the pair as one
rename: the copy's message SHALL say the source was deleted and SHALL name
`taskless rule restore <copyOf.ruleId>`, and the CLI SHALL NOT print a separate `missing`
warning for the source. A runtime rename SHALL be one notice and SHALL NOT change the exit
code. `copyOf` absent or `null` SHALL be treated as no copy. A `copyOf` that is present but
has no non-empty string `ruleId` SHALL fail closed: the rule SHALL be treated as
unaccounted.

#### Scenario: A renamed and loosened Vale rule fails check as one rename

- **WHEN** reconcile returns vale rule `bar-2` in `unknown` with `copyOf.ruleId` `foo-1` and `copyOf.files` listing `.vale.ini` changed, and returns `foo-1` as `missing`
- **THEN** `bar-2` SHALL NOT run
- **AND** `check` SHALL exit non-zero with one message saying `bar-2` is a copy of `foo-1`, which was deleted, naming `.vale.ini` as changed and `taskless rule restore foo-1`
- **AND** the CLI SHALL NOT print a separate warning that `foo-1` is missing

#### Scenario: A copy beside its present source fails check

- **WHEN** reconcile returns sg rule `bar-2` in `unknown` with `copyOf.ruleId` `foo-1`, and `foo-1` is not `missing`
- **THEN** `bar-2` SHALL NOT run
- **AND** `check` SHALL exit non-zero naming `bar-2` as a copy of `foo-1`

#### Scenario: A runtime copy is not executed and does not fail

- **WHEN** reconcile returns a runtime rule in `unknown` with `copyOf`
- **THEN** the rule SHALL NOT execute
- **AND** its skip reason SHALL name the source rule
- **AND** the exit code SHALL NOT change because of it

#### Scenario: An unreadable copyOf fails closed

- **WHEN** reconcile returns a rule in `unknown` whose `copyOf` is present but has no string `ruleId`
- **THEN** the rule SHALL NOT run or execute
- **AND** `check` SHALL exit non-zero naming it
Loading
Loading