Skip to content

Add provenance-linked real-project and minimized Workshop regression evidence #20

Description

@Teakowa

Parent: #10
Depends on: #18

Goal

Add real Workshop project evidence and provenance-linked minimized regressions so canonical correctness claims are grounded in externally meaningful behavior rather than only synthetic fixtures or implementation-local snapshots.

Context

workshop-rs already has focused parser/catalog/emitter/round-trip tests. #10 requires a distinct evidence layer where real project failures remain reproducible at both project and minimized-feature granularity, with explicit provenance and independently grounded expectations.

Scope

  • Define the repository layout and metadata required for preserved real-project Workshop evidence and minimized regressions under the Define the Workshop conformance result contract and stable feature identities #18 provenance/result contract.
  • Add at least one reviewed real-world Workshop project/revision/path as corpus evidence, respecting repository licensing/provenance policy.
  • Identify at least one meaningful behavior or failure from real-project evidence and preserve a minimized reproduction linked back to the original project provenance.
  • Keep the complete project-level case in addition to the minimized case so cross-rule, variable, settings, localization, and other integration interactions remain testable.
  • Attribute regressions to stable Define the Workshop conformance result contract and stable feature identities #18 Workshop feature identities where applicable.
  • Ensure expected behavior is independently supported by preserved source/client/reference evidence and is not established solely by snapshotting the current implementation output.
  • Add corpus execution/reporting that distinguishes expected matches, known gaps, unsupported cases, regressions/divergence, and inconclusive evidence.
  • Document how future real-project regressions should be admitted, minimized, reviewed, and retained.

Non-goals

  • Replacing focused unit/property/census tests with real-project tests.
  • Importing OPY or DEL/OSTW source projects as canonical Workshop semantics.
  • Bulk-vendoring unreviewed community projects.
  • Treating current workshop-rs output as an independent oracle.
  • Requiring a live-client automation system before real-project evidence can land.

Acceptance criteria

  • At least one real Workshop project case is preserved with repository/revision/path and licensing/provenance metadata.
  • At least one project-derived behavior/failure has a minimized regression linked to the original evidence.
  • Full-project and minimized cases remain separate complementary test layers.
  • Conformance results use Define the Workshop conformance result contract and stable feature identities #18 states/feature identities and cannot report known failures as successful compatibility.
  • The repository documents a repeatable evidence-admission/minimization process suitable for future regressions.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions