feat(conformance): define Workshop evidence contract - #26
Conversation
Define stable locale-independent feature identities and machine-readable conformance results with explicit evidence provenance, comparison states, and validation rules. Document the public contract for future census, corpus, and live-client workflows.
Separate fixture and expectation provenance, namespace feature identities, add structured comparison and reason data, and validate artifacts through the JSON entry point. Fixes #18
Architect review\n\nIndependent Architect review identified four contract risks before dependent work: unscoped feature names, fixture/expectation provenance conflation, free-text-only result semantics, and raw JSON deserialization bypassing validation. Commit 0bf6917 addresses these by adding namespace-qualified identities (including domain-qualified enum members), separate fixture/expectation artifacts, structured comparison/reason codes with digest checks, and ConformanceResult::from_json validation. The review boundary remains explicit: this PR establishes the contract only; #19/#20/#21 must consume it without redefining it. No Architect approval is being claimed here; maintainer review remains required. |
Require matched expected and observed artifacts to be distinct from the fixture and recorded implementation output, and cover the false-positive path with a regression test. Fixes #18
|
Fresh QA found an artifact-aliasing false-positive risk. Commit 68a4340 now rejects matched expected/observed artifacts that are identical to each other, the executed fixture, or the recorded implementation artifact, with a regression test. This was revalidated locally with focused conformance tests, clippy -D warnings, catalog check, and diff-check; full CI is required again at the new head. |
|
QA follow-up addressed in commit 0e2ee3d: validate_against(Catalog) now rejects fabricated catalog and enum-member identities; comparison artifacts remain valid when non-materialized (optional SHA-256); ADR-0002 explicitly assigns execution of semantic/normalized comparison to the independent #19 gate. Local full workspace tests, clippy, catalog check, fmt, and diff-check pass. Awaiting fresh CI and QA review at this head. |
Reject expected artifacts that alias the implementation output while preserving the valid expectation-source relationship, with explicit negative coverage. Fixes #18
|
Fresh QA blocker resolved in commit 9a1f753: expected artifacts are now also rejected when they alias the implementation artifact, while expected == expectation.artifact remains the valid independent-oracle relationship. Added regression coverage. Current local conformance tests/clippy/catalog check pass; awaiting CI and final independent QA. |
Fresh independent QA — VERIFIED / PASSExact head reviewed:
The previous QA on |
Summary
Implements #18 by adding the canonical
workshop_rs::conformancepublic contract.FeatureIdvalues derived from canonical catalog identities or workshop-rs-owned namespaces.Evidence and checks
The contract has focused positive/negative serialization and validation tests, including duplicate features, status/reason mismatch, missing live-client provenance, missing artifact digests, implementation metadata not serving as an oracle, and a format-valid mismatched catalog-digest regression.
ConformanceResult::validate_againstrequires the completeevidence.catalog == supplied Catalog::identity()before accepting catalog-backed feature identities.cargo fmt --all --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspace --all-targetscargo run -p workshop-rs --bin workshop-catalog-gen -- checkgit diff --checkThis PR establishes the offline contract only. It does not claim census coverage, real-project evidence, or live-client captures; those are the dependent #19–#21 workflows.