Software factory: validator runs that check zero files no longer pass

[jurgenwerk]

Fixes CS-11426.

A validator that says "pass" should mean "I looked at your code and it's fine." Before this fix, it could also mean "I found no code to look at" — and it said "pass" either way. So the agent would ask "is everything OK?" before its files had actually arrived on the realm, get a "pass" that checked nothing, and move on believing its work was verified when nothing was. Now that situation returns an error saying "there was nothing to check — re-run," instead of a fake green light.

Both footguns were observed in real adjust-flow runs: the first baseline run_parse returned {status: "passed", filesChecked: 0} because it ran before the seeded files reached the realm, and a single-file run_parse on a component false-failed with cross-file resolution errors a whole-realm run contradicted.

What

Vacuous passes are now errors (tool layer). A whole-realm run_parse / run_lint / run_evaluate / run_instantiate / run_tests that passed while checking zero files / modules / instances / tests is rewritten to status: "error" with a message telling the agent to re-run once the realm contains the files. Until now the zero-coverage rule existed only as instruction text ("a zero-test run counts as failed") with no enforcement anywhere. Single-file runs are exempt — an explicit path either resolves to one checked file or errors on its own.

run_parse and run_lint now sync the workspace first, exactly like the other three validators already did. Their engines read realm files, so an unsynced workspace was invisible to them — that's the precise mechanism behind the observed filesChecked: 0 false green. With all five syncing, the seed sequence (copy → sync → validate) is enforced mechanically rather than by prose.

Scope choice: the rewrite lives in the agent-facing tool wrappers, not the shared engines. The validation pipeline's step semantics ("no files returns passed", asserted by name in the step tests) are unchanged — bootstrap issues use the no-op validator and implementation issues always have files by validation time.

Docs: the bootstrap-seed adjust instructions and the orchestrator operations skill state both behaviors, and the skill now documents the single-file run_parse footgun (per the ticket's option b): files importing same-realm siblings can false-fail in isolation; whole-realm parse is the source of truth.

Verification

• 8 new unit tests (vacuous rewrite per tool, single-file exemption, pass-through of real coverage and of genuine errors, sync-before-engine ordering, sync-failure surfacing); factory-tool-builder.test.ts 44/44 green.
• validation-pipeline.test.ts, issue-loop.test.ts, factory-brief.test.ts all green locally; ember-tsc, eslint, prettier clean.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR strengthens Software Factory validator semantics so that a “passed” validation result always implies real work was performed, preventing false-green baseline gates when the realm/workspace isn’t yet populated or indexed.

Changes:

• Rewrite whole-realm run_* results from passed → error when they report zero checked files/modules/instances/tests (single-file path runs are exempt).
• Add workspace sync-before-run for run_parse and run_lint (bringing them in line with the other validators).
• Update adjust-flow seed instructions + orchestrator operations skill docs, and add unit tests covering the new behaviors.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
packages/software-factory/src/factory-tool-builder.ts	Adds vacuous-pass rewriting to error results; adds sync-before-run for run_parse and run_lint; updates tool descriptions accordingly.
packages/software-factory/tests/factory-tool-builder.test.ts	Adds unit tests for vacuous-pass rewriting and sync-before-engine ordering/failure surfacing.
packages/software-factory/src/factory-seed.ts	Updates adjust-seed instructions to reflect sync + zero-coverage-is-error behavior.
packages/software-factory/.agents/skills-orchestrator/software-factory-operations/SKILL.md	Updates ops skill docs to reflect validator sync behavior and zero-coverage semantics; documents the single-file run_parse caveat.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.