Possible Missing Test Migrations

[theCodeDrift]

The current 3=>5 migration may miss migrating tests

I had to give no-eval a .tests/ directory. It shipped without one, and ast-grep's test discovery walks every rule dir — so test failed on the missing directory before it ever reached my rule. It now has 3 valid / 3 invalid cases and passes. Without this, test was broken for any rule you add.

npx @taskless/cli-nightly@0.11.0-20260824013916xe788b11

[theCodeDrift]

Confirmed and reproduced locally. Nightly-only — migrations 0004/0005 have never been in a stable release (latest published is 0.10.2), so this affects @taskless/cli-nightly installs.

Root cause

Migration 0005 creates a rule's .tests/ only as a side effect of moving a test file into it:

• moveEngineRules (packages/cli/src/filesystem/migrations/0005-rule-directories.ts:90-128) creates rules/sg/<id>/ and moves <id>.yml. It never touches .tests/.
• moveEngineTests (:137-178) creates .tests/ only for a file matching ^(?<id>.+?)-\d{8}-test\.ya?ml$ (:163) and continues past anything else.

So any sg rule that had no test file at version 3 — or one whose filename lacked the -YYYYMMDD-test.yml timestamp — arrives at version 5 as a rule directory with no .tests/.

Meanwhile assembleSgConfig (packages/cli/src/rules/assemble.ts:141-164) emits a testDir entry for every rule id, existence unchecked. ast-grep 0.41.0 then refuses the entire invocation:

Error: Cannot read rule directory .taskless/rules/sg/no-eval/.tests
╰▻ IO error ... No such file or directory (os error 2)     # exit 6

--filter does not scope this. taskless test .taskless/rules/sg/<some-other-rule> fails with the same error, which is exactly why adding a correct new rule doesn't help — the failure names a rule the author never touched.

check is unaffected (sg scan ignores testConfigs), and so is verify (it doesn't run layer 3 — commands/verify.ts:126-143).

sg-specific. Vale reads a missing .tests/ as empty (rules/vale/verify.ts:53-62); runtime has no per-rule test directory.

Reproduction

Seed a v3 project with one rule that has a timestamped test and one that has none, then run test:

✗ sg/no-eval    No test file found for rule "no-eval" in .taskless/rules/sg/no-eval/.tests/
✗ sg/no-var     Error: Cannot read rule directory .taskless/rules/sg/no-eval/.tests
2 of 2 rule(s) failed.

Measured against ast-grep 0.41.0 for the fix space: an empty .tests/ is fine (1 passed; 0 failed, exit 0), a .tests/.gitkeep is fine for both sg test and sg scan, and an empty testConfigs list is fine.

Fix

A new migration 0006 that ensures .tests/ for every rules/sg/<id>/, with a .gitkeep inside. The .gitkeep is load-bearing rather than cosmetic: git does not track empty directories, so without it the repair never reaches CI or a fresh clone and the bug returns there. 0004 (ensureTrackedDirectory, :322-328) and 0005 (scaffoldEngineDirectories, :189-198) already do this.

It must be a new migration rather than an amended 0005: runMigrations short-circuits on version === maxVersion (filesystem/migrate.ts:206-208), so a manifest already stamped "version": 5 by a nightly — the reporter's exact state — would never re-run an amended 0005. Bumping maxVersion to 6 is what reaches those installs. Prototyped and verified for both the v3→6 path and the already-at-v5 repair.

Worth landing alongside it, because the migration is only one route into this state:

1. assembleSgConfig should skip a testDir that isn't on disk. A hand-created mkdir .taskless/rules/sg/foo hits the identical hard failure with no migration involved, and one rule should never be able to fail every other rule's test run. verify/test still report the rule honestly ("No test file found" / "Skipped: no test file found"), so nothing becomes a silent pass.
2. Ingest path has the same gap: commands/rules.ts:244 and :484 gate writeRuleTestFile behind if (rule.tests), so an API-delivered rule with no tests also produces a rule directory with no .tests/.

The general class

Same shape elsewhere, lower severity — these soft-fail rather than crashing, but they are the same "scaffold guarantees it, migration doesn't" gap:

• Vale .vale.ini — 0005:246-309 writes a per-rule config only for matchers carrying a tskl) rule = <id> breadcrumb. A hand-written matcher yields a rule directory with no .vale.ini; assembleValeConfig (assemble.ts:104-115) skips it silently and the rule is disabled. 0005's own doc comment (:342-350) dismisses this case as unreachable because 0004 is unreleased — an assumption that is weaker now that nightly users exist.
• Runtime captures/ — 0005:110-117 creates it only when there is a loose *.yml to move. Reported as an error by inspect.ts:175-188.
• Cosmetic: .taskless/sg/ survives as an empty directory, because pruneEmpty (0005:362-366) runs before the PRIOR_CONFIGS removal at :368-371, so sg/ still contains sgconfig.yml when it's inspected.

Coverage gap

No existing test catches this, and the suite comes close enough to be worth noting. test/migrate-engine-layout.test.ts:78 already seeds an untimestamped rule-tests/no-eval-test.yml, and the test at :107-109 comments that 0005 "cannot attribute it and leaves it behind" — that fixture is this bug. Nothing then asserts the resulting tree is usable. test/assemble.test.ts:34-43 can't reach the case at all, because its sgRule helper always creates the tests directory.

Two assertions close it: every emitted testDir exists (in assemble.test.ts, no subprocess needed), and every rules/sg/*/ has a .tests/ after migration (on the fixture that already reproduces it).