Skip to content

Collapse OPY compatibility evidence metadata into ordinary differential tests #337

Description

@Teakowa

Goal

Simplify OPY compatibility verification into conventional differential/regression tests and test data, removing evidence-specific metadata and historical task bookkeeping that are not needed to execute or understand the tests.

Context

The current OPY differential suite is meaningful and should remain: it runs OPY inputs through the native implementation and compares supported observable behavior with the pinned OverPy reference.

However, current main layers additional evidence-specific state around those tests:

  • crates/opy-rs/tests/differential.rs contains a large declared-corpus table with Issue-specific rationale/history in test expectations;
  • differential-expectations.json defines an evidencePolicy and per-case evidence arrays in addition to native expectations;
  • each corpus case carries fixture.json plus oracle.json, with overlapping status/reference/provenance information;
  • current docs/ADR-0002 describe an "evidence-first" compatibility architecture rather than ordinary differential tests;
  • support/current documentation routes developers through corpus/oracle/evidence terminology as a distinct model.

The desired model from wrightkit/.github#60 is: OPY compatibility is a test requirement; OverPy is the reference implementation; committed expected outputs/reference snapshots are test data; third-party provenance is minimal and practical.

Scope

  • Keep the pinned OverPy reference runner and differential comparison where they materially protect OPY behavior.
  • Make the differential test itself the primary executable compatibility contract.
  • Collapse duplicated expectation/status metadata so each test case has one clear expected native/reference relationship.
  • Remove generic evidencePolicy, evidence arrays, Issue/PR/task references, and historical implementation narrative from committed test expectations.
  • Simplify fixture.json / oracle.json or replace them with more direct test-data forms where the current schema does not serve an active test requirement.
  • Preserve only the provenance needed for redistributed upstream/third-party fixtures, licensing, immutable identity, and reproducible reference comparison.
  • Keep generated/run reports under target/ or CI artifacts rather than committing point-in-time execution results.
  • Reconcile AGENTS, language-support, upstream-reference, compatibility-manifest, architecture, and corpus documentation with the tests-first model.
  • Supersede or partially supersede ADR-0002 where it establishes evidence-first architecture; preserve the historical ADR rather than rewriting history.
  • Reconcile open parent/support contracts that still require "compatibility evidence" as a distinct subsystem.

Non-goals

  • Removing meaningful compiler, semantic, differential, real-project, or negative compatibility tests.
  • Removing the pinned OverPy reference as a behavioral oracle/reference implementation.
  • Requiring byte-identical Workshop output where observable compatibility does not require it.
  • Removing licensing/redistribution metadata required for imported fixtures.
  • Moving OPY compatibility ownership to Wright or workshop-rs.
  • Replacing the current metadata layers with another generic test framework.

Acceptance Criteria

  • A contributor can understand OPY compatibility as input + native execution + reference/expected result + assertion.
  • Differential tests retain the same meaningful failure detection after metadata simplification.
  • There is one authoritative expected relationship per test case rather than overlapping expectation tables and evidence records.
  • Current test data contains no Issue/PR/task history unless that identifier is itself part of the tested public behavior.
  • Third-party fixture metadata is limited to attribution/licensing/immutable identity/reproducibility needs.
  • Current docs no longer require an evidence-first mental model.
  • ADR history remains truthful while current contracts point to tests-first compatibility verification.
  • Wright does not gain duplicate OPY compatibility ownership as a result of the cleanup.

Dependencies / Ownership

ADR cleanup requirement

ADR cleanup is part of this Issue, not a later documentation task.

Current ADR audit found:

  • ADR-0002 explicitly establishes an "evidence-first" compatibility model and must be superseded or partially superseded by the tests-first decision;
  • ADR-0001/0003/0004/0005 and the ADR registry still use "historical evidence", "compatibility evidence", "conformance evidence", or similar terminology even where the durable architecture decision is otherwise valid.

Required treatment:

  • add a current tests-first compatibility/testing ADR/decision when needed to replace ADR-0002's evidence-first contract;
  • mark ADR-0002 Superseded/Partially Superseded and remove it from current routing authority;
  • preserve the historical ADR body, but make the supersession explicit at the top and in the ADR registry;
  • update every remaining Accepted ADR so current decision language refers directly to tests, differential/reference comparison, source provenance, diagnostics/provenance, licensing, or reproducibility rather than generic evidence;
  • rename "Historical evidence" fields/references to related/decision history;
  • keep OverPy reference pinning and source provenance where they remain real compatibility/licensing contracts.

Acceptance is not satisfied while any Accepted/current OPY ADR declares an evidence-first model or uses generic evidence as an architectural authority.

Terminology requirement: no generic provenance abstraction

This cleanup also removes provenance as a generic WrightKit domain concept.

Use the concrete concern instead:

  • source mapping / source location for diagnostics, edits, or generated-to-authored attribution;
  • source attribution for imported or redistributed test/source data;
  • license / licensing metadata for legal and redistribution requirements;
  • pinned revision / version for immutable external references;
  • checksum / hash only where integrity or reproducibility requires it;
  • reproducibility for the commands/configuration/version information needed to recreate a current test/reference result.

Do not preserve generic concepts such as provenance-aware fixture, provenance-linked regression, diagnostic provenance, catalog provenance, artifact provenance, or a generic provenance schema merely to group unrelated metadata. If a field or abstraction cannot be mapped to a concrete continuing requirement above, remove it.

This does not authorize removing source spans/mappings, third-party attribution, licensing metadata, or immutable reference identity where those are actually required.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions