refactor: adopt tests-first Workshop verification - #242
Conversation
Remove the evidence taxonomy from conformance, corpus, census, and capture contracts while preserving direct expected results, source provenance, and client metadata. Update fixtures and ADR routing for the tests-first model. Fixes #241
Teakowa
left a comment
There was a problem hiding this comment.
Requesting changes for three blocking #241 acceptance-criteria gaps:
-
crates/workshop-rs-cli/src/corpus.rs:207-226still does not execute the declared reference comparison. The runner validates the expected artifact's metadata, computes a digest for the observed WIR dump, then returnsMatchedwhenever parse/WIR/catalog validation succeeds. A wrong or unrelated oracle can therefore still produce a matched result. This does not preserve an independent expected result or meaningful real-project regression coverage. Required fix: make the expected result participate in an actual comparison (or stop representing the case as a matched reference comparison), and add a negative regression proving that a deliberately divergent expected result cannot pass. -
#241 explicitly requires the still-open #10 contract to stop directing future work toward an evidence taxonomy/result architecture. Live #10 still requires an evidence model, provenance-aware fixture classes, expectation/evidence sources, and evidence-specific acceptance criteria. Required fix: reconcile #10's durable contract with the tests-first model before #241 is considered complete.
-
The current ADR cleanup is incomplete. For example, Accepted ADR-0007 still defines an "evidence lifecycle",
EvidenceRef, "topology evidence", and dataset evidence as current decision language; other current Accepted ADRs still retain generic provenance/evidence wording despite #241 requiring concrete terms such as source attribution, license, pinned revision/version, source mapping, or runtime validation. Required fix: finish the review of all Accepted/current ADRs and current contracts so the surviving guidance no longer teaches the generic evidence/provenance model. Historical text in ADRs that are fully/partially superseded can remain as history when the supersession is unambiguous.
Do not report a matched corpus result when the pinned oracle output is not available for comparison. Add a divergent-reference regression and finish the current tests-first source terminology cleanup. Refs #241
|
Addressed all three requested changes in f11baaf:
Local format, clippy, and workspace all-target tests pass. |
Teakowa
left a comment
There was a problem hiding this comment.
One blocking follow-up remains from finding 3:
docs/adr/0001-catalog-boundaries.mdstill defines a generic Provenance contract ("Every catalog entry and locale table carries provenance" grouping source identity, license/review status, and generator identity), and live #10 still says "provenance-linked regression fixtures". #241 explicitly says not to preserve generic provenance abstractions or that exactprovenance-linked regressionconcept; use the concrete continuing concerns instead.
Required fix: replace the remaining current-contract umbrella wording with the actual fields/requirements (source attribution/identity, license/review metadata, pinned revision/hash, generator/reproducibility information as applicable), and remove the prohibited provenance-linked regression wording from #10. Historical superseded ADR text can remain unchanged.
Replace the remaining generic provenance umbrella in ADR-0001 with source identity, attribution, license/review metadata, and pinned revision or hash requirements. Refs #241
|
Addressed the remaining provenance wording follow-up in e824f3a:
The branch is clean and pushed at e824f3a. |
Summary
Verification
Fixes #241