diff --git a/AGENTS.md b/AGENTS.md index 5ce54727..acd1bc6c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,11 +13,11 @@ or LPP, while `opy-rs` must remain independently usable as a library and CLI. `opy-rs` owns: - OverPy syntax, lexer, parser, source model, preprocessing, macros, - semantic resolution, diagnostics, provenance, and OPY HIR; + semantic resolution, diagnostics, source mapping, and OPY HIR; - OverPy-specific compiler/lowering semantics and backend-affecting behavior; - standalone OPY tooling APIs and CLI surfaces; - Workshop → OPY reconstruction when implemented; -- OPY compatibility evidence and support claims. +- OPY compatibility tests and support claims. `workshop-rs` owns: @@ -38,7 +38,7 @@ into this repository. Never add a dependency from `workshop-rs` back to The standalone source-analysis path must remain usable without requiring Workshop emission. Compilation may depend on `workshop-rs`; `check`, semantic inspection, source queries, and other Workshop-independent operations should -not be forced through the compiler pipeline without an evidence-backed need. +not be forced through the compiler pipeline without a concrete requirement. Do not invent WrightKit-only OPY syntax. Compatibility is observable semantics, not output-text identity, optimizer implementation, formatting, temporary @@ -63,7 +63,7 @@ If the Issue, current contract, and source/tests disagree materially, stop and surface the mismatch rather than selecting a design by implementation convenience. -## Upstream reference and provenance +## Upstream reference and source attribution For the declared OverPy core-language surface, the established upstream OverPy implementation is the executable specification. Core behavior is presumptively @@ -72,11 +72,11 @@ or demonstrated to be a non-contractual implementation artifact. Inspect upstream implementation, docs, and tests to understand behavior; then implement the behavior directly in clear Rust and verify it through -compatibility evidence. Do not mechanically translate, import, link, or bundle +compatibility tests and reference comparisons. Do not mechanically translate, import, link, or bundle upstream implementation/data into the `opy-rs` core or release artifacts. OverPy is GPL-3.0; this repository is AGPL-3.0-or-later. Pinned identity, -fixture/probe provenance, and licensing boundaries are documented in +fixture/probe attribution, and licensing boundaries are documented in [`docs/compatibility/upstream-references.md`](docs/compatibility/upstream-references.md). Inventories, manifests, probes, differential tests, corpus fixtures, and real @@ -89,7 +89,7 @@ Prefer typed Rust for observable OverPy behavior and invariants: receiver/member semantics, argument binding, contextual dispatch, macro/directive behavior, coercion/evaluation rules, and special lowering. -Declarative data may carry large mechanical inventories, names, provenance, and +Declarative data may carry large mechanical inventories, names, source attribution, and facts that do not themselves program language behavior. Do not extend an existing manifest/registry with new semantic control fields merely because the current implementation already contains similar metadata; existing @@ -104,7 +104,7 @@ project exposes a blocker: 2. fix it here if the missing behavior is OverPy-owned; 3. route missing canonical Workshop behavior to `workshop-rs`; 4. preserve a minimized regression where practical while keeping the full - project evidence; + project test coverage; 5. do not split implementation work into smaller issues solely for bookkeeping when one coherent change can be reviewed and validated safely. @@ -122,8 +122,8 @@ cargo test --workspace --all-targets --all-features python3 -m unittest discover -s tools/overpy/tests ``` -Oracle-required compatibility probes run separately against the pinned -reference. A local test count is not sufficient evidence when a change claims +Oracle-required compatibility tests run separately against the pinned +reference. A local test count is not sufficient when a change claims real-project compatibility; rerun the affected full-project workflow. ## Delivery @@ -132,6 +132,6 @@ real-project compatibility; rerun the affected full-project workflow. through PRs. - Keep commits focused and issue-linked where an issue exists. - Keep compatibility/support claims synchronized with actual implementation and - evidence. + tests. - Never commit credentials, private runtime data, or unreviewed third-party material. diff --git a/README.md b/README.md index 47cbdc7e..04f2d079 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ or as a Language Provider Protocol (LPP) process for extended linting, analysis, and editor support. `opy-rs` owns OverPy syntax, preprocessing, macros, semantic resolution, -compiler lowering, diagnostics, provenance, and source reconstruction. Shared +compiler lowering, diagnostics, source mapping, and source reconstruction. Shared Workshop semantics, catalog identities, and emission remain delegated to `workshop-rs`. @@ -84,7 +84,7 @@ or upstream internal architecture. Optimizer behavior that materially changes Workshop structure or element cost remains in scope; only incidental optimizer shape is outside the target. Support claims are backed by the canonical [language-support contract](docs/language-support.md), the corpus, -and pinned OverPy reference evidence. +and pinned OverPy reference results. > [!IMPORTANT] > `opy-rs` follows the OverPy language. It does not introduce a WrightKit-only @@ -118,7 +118,7 @@ Python test suites. ## Documentation -Current architecture, compatibility evidence, APIs, HIR, provenance, and +Current architecture, compatibility tests, APIs, HIR, source mapping, and maintainer references are indexed in [`docs/README.md`](docs/README.md). ## Contributing diff --git a/crates/opy-cli/src/bin/opy-compat.rs b/crates/opy-cli/src/bin/opy-compat.rs index 64fb641e..fd73cda3 100644 --- a/crates/opy-cli/src/bin/opy-compat.rs +++ b/crates/opy-cli/src/bin/opy-compat.rs @@ -1,4 +1,4 @@ -//! Internal compatibility evidence producer. +//! Internal compatibility comparison producer. //! //! This target is built only by the compatibility gate. It is deliberately //! separate from `opy-cli compile`: pinned oracle input must never become part diff --git a/crates/opy-rs/src/compiler/tests/resource_baseline.rs b/crates/opy-rs/src/compiler/tests/resource_baseline.rs index ed2f885e..0c037074 100644 --- a/crates/opy-rs/src/compiler/tests/resource_baseline.rs +++ b/crates/opy-rs/src/compiler/tests/resource_baseline.rs @@ -61,7 +61,7 @@ struct InputIdentity { sha256: String, source: String, parameters: JsonValue, - provenance: JsonValue, + attribution: JsonValue, } #[derive(Debug)] @@ -356,7 +356,7 @@ fn synthetic_identity(id: &str, source: String, parameters: JsonValue) -> InputI sha256: sha256(&source), source: format!("compiler/tests/resource_baseline.rs::{id}"), parameters, - provenance: serde_json::json!({ + attribution: serde_json::json!({ "kind": "original", "origin": "WrightKit opy-rs resource audit workload", "license": "AGPL-3.0-or-later", @@ -376,9 +376,9 @@ fn real_world_workload() -> Workload { .expect("parabola metadata is valid JSON"); Workload { id: "real-world-parabola", - description: "provenance-linked OverPy parabola example compiled end to end", + description: "attributed OverPy parabola example compiled end to end", input: InputIdentity { - kind: "provenance-linked-corpus".to_string(), + kind: "attributed-corpus".to_string(), bytes: source.len(), sha256: sha256(&source), source: format!("{REAL_FIXTURE}/parabola.opy"), @@ -389,7 +389,7 @@ fn real_world_workload() -> Workload { ), "entry": "parabola.opy" }), - provenance: metadata["provenance"].clone(), + attribution: metadata["attribution"].clone(), }, run: run_real_world, } diff --git a/crates/opy-rs/tests/differential-expectations.json b/crates/opy-rs/tests/differential-expectations.json deleted file mode 100644 index 0c498159..00000000 --- a/crates/opy-rs/tests/differential-expectations.json +++ /dev/null @@ -1,84 +0,0 @@ -{ - "schemaVersion": 1, - "description": "Independent native outcome expectations for the OPY differential corpus.", - "evidencePolicy": { - "reference": "fixture.json expectedStatus plus the pinned oracle.json snapshot", - "native": "An implementation-local invariant or an explicitly recorded known gap; native behavior is never used to create reference evidence.", - "classification": "match, known-gap, and unsupported are expected relationships. unexpected-divergence, regression, and inconclusive are produced by the runner." - }, - "cases": [ - {"fixture": "synthetic/array-unique", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/array-unique/oracle.json", "provenance:synthetic/array-unique/fixture.json", "test:opy-rs::compiler::integration_tests::array_unique::array_unique_matches_the_pinned_canonical_wir"], "note": "Array.unique() resolves as an OPY member and preserves first-occurrence order through canonical filtered-array lowering."}, - {"fixture": "synthetic/basic-rule", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/basic-rule/oracle.json", "implementation-invariant:source implementation-resolves-basic-rule"], "note": "Minimal rule is part of the Workshop-independent source implementation contract."}, - {"fixture": "synthetic/control-flow", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/control-flow/oracle.json", "implementation-invariant:source implementation-resolves-control-flow"], "note": "If/elif/else, for-in-range, while, and pass resolve in the OPY semantic model."}, - {"fixture": "synthetic/syntax-surface", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/syntax-surface/oracle.json", "implementation-invariant:syntax-hir"], "note": "Switch, do-while, hexadecimal literals, membership, dict indexing, comprehensions, lambda arguments, and f/w/b/c string modifiers resolve through the OPY HIR."}, - {"fixture": "synthetic/string-modifiers", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/string-modifiers/oracle.json", "implementation-invariant:string-modifier-hir"], "note": "The pinned oracle accepts the inventory-backed f/w/b/c modifier forms; l/t remain syntax-carried because translation content is outside this source implementation."}, - {"fixture": "synthetic/switch-break", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/switch-break/oracle.json", "implementation-invariant:switch-fallthrough-hir"], "note": "The pinned oracle accepts source-order switch fallthrough and nested break; native HIR retains the same authored arm order and explicit break statements."}, - {"fixture": "synthetic/strings-and-lambda", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/strings-and-lambda/oracle.json", "implementation-invariant:format-provenance-and-lambda-slots"], "note": "The pinned oracle accepts semantic f-string interpolation and sorted key lambda syntax; native HIR retains format arguments with source provenance."}, - {"fixture": "synthetic/lambda-invalid", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/lambda-invalid/oracle.json", "implementation-invariant:lambda-context-diagnostic"], "note": "The pinned oracle rejects a standalone lambda argument; native emits the stable lambda-context diagnostic."}, - {"fixture": "synthetic/syntax-invalid", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/syntax-invalid/oracle.json", "implementation-invariant:structured-negative-diagnostics"], "note": "Malformed do-while and dictionary syntax remain structured failures rather than silent acceptance."}, - {"fixture": "synthetic/declarations-numbers", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/declarations-numbers/oracle.json", "implementation-invariant:source implementation-resolves-declarations"], "note": "Numeric literals and variable-index declarations resolve in the OPY semantic model."}, - {"fixture": "synthetic/declarations-rules", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/declarations-rules/oracle.json", "implementation-invariant:source implementation-resolves-rule-declarations"], "note": "globalvar, playervar, subroutine, def, enum, and rule declarations resolve."}, - {"fixture": "synthetic/expressions-values", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/expressions-values/oracle.json", "implementation-invariant:source implementation-resolves-expressions"], "note": "Expressions, arrays, strings, vectors, calls, and format expressions resolve."}, - {"fixture": "synthetic/preprocessing", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/preprocessing/oracle.json", "implementation-invariant:preprocessor-include-define"], "note": "Include, object-like and function-like defines are preserved through preprocessing."}, - {"fixture": "synthetic/directives", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/directives/oracle.json", "implementation-invariant:directive-preprocessing-state"], "note": "Advanced directive state and source annotations resolve without executing Workshop optimizer, localization, or emission effects."}, - {"fixture": "synthetic/duplicate-rule-diagnostic", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/duplicate-rule-diagnostic/oracle.json", "implementation-invariant:duplicate-rule-name-frontier"], "note": "The ordinary-rule @Name form reaches the pinned duplicate-rule-name semantic frontier with source attribution."}, - {"fixture": "synthetic/project-main-file", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/project-main-file/oracle.json", "implementation-invariant:project-main-file-scope"], "note": "mainFile redirects the source implementation entry point and preserves child-include prefix state without copying catalog data."}, - {"fixture": "synthetic/project-entry", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/project-entry/oracle.json", "provenance:synthetic/project-entry/fixture.json"], "note": "The pinned oracle accepts a leading included-file #!mainFile directive, while native compilation preserves the declared root entry point and both rule outputs."}, - {"fixture": "synthetic/legacy-entry-include", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/legacy-entry-include/oracle.json", "provenance:synthetic/legacy-entry-include/fixture.json"], "note": "The minimized entry topology preserves OverPy's latest resolved file lookup base and source file provenance."}, - {"fixture": "synthetic/directives-scoped", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/directives-scoped/oracle.json", "implementation-invariant:scoped-directive-surfaces"], "note": "Pinned positive probe covers global rulePrefixTemplate, include prefix restoration, AST macro/enum redeclaration, and translation normalization."}, - {"fixture": "synthetic/translations-invalid", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/translations-invalid/oracle.json", "implementation-invariant:pinned-translation-set"], "note": "An outside-code translation is rejected by both the pinned oracle and native source implementation."}, - {"fixture": "synthetic/include-scope", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/include-scope/oracle.json", "implementation-invariant:scoped-preprocessing"], "note": "Nested include optimization directives remain observable as scoped preprocessing events; strict-sensitive lowering is covered by the compiler compatibility contract."}, - {"fixture": "synthetic/optimize-strict", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/optimize-strict/oracle.json", "provenance:synthetic/optimize-strict/fixture.json"], "note": "The native frontend accepts the strict optimizer probe and preserves its source rule for compiler-level semantic-WIR verification."}, - {"fixture": "synthetic/settings", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/settings/oracle.json", "implementation-invariant:settings-structure-only"], "note": "Settings are structurally represented; Workshop key and leaf validation remains lowering-dependent."}, - {"fixture": "synthetic/receiver-calls", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/receiver-calls/oracle.json", "implementation-invariant:receiver-call-resolution"], "note": "The exercised receiver/member forms resolve through the OPY semantic model."}, - {"fixture": "synthetic/receiver-playervar", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/receiver-playervar/oracle.json", "implementation-invariant:receiver-playervar-member-expression"], "note": "A bare variable member expression is retained as an OPY HIR member node; canonical Workshop member validation remains lowering-dependent."}, - {"fixture": "synthetic/chase-enums", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/chase-enums/oracle.json", "implementation-invariant:opaque-workshop-enum-identities"], "note": "Declared chase enum identities resolve without duplicating Workshop catalog validation."}, - {"fixture": "synthetic/chase-condition-agentlab", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/chase-condition-agentlab/oracle.json", "regression:agent-lab-chase-condition-shape"], "note": "The agent-lab chase condition shape is retained as an OPY source implementation regression."}, - {"fixture": "synthetic/chase-keywords", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/chase-keywords/oracle.json", "implementation-invariant:keyword-binding"], "note": "Generic keyword binding and the chase contextual form resolve."}, - {"fixture": "synthetic/for-range-agentlab", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/for-range-agentlab/oracle.json", "regression:agent-lab-for-range-binder"], "note": "Implicit for-range binders are retained as an OPY source implementation regression."}, - {"fixture": "synthetic/member-angle", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/member-angle/oracle.json", "provenance:synthetic/member-angle/fixture.json", "test:opy-rs::compiler::integration_tests::member_angles::horizontal_facing_angle_member_matches_the_pinned_canonical_wir"], "note": "The catalog-backed eventPlayer.getHorizontalFacingAngle() member value resolves through the OPY manifest and matches the pinned canonical WIR."}, - {"fixture": "synthetic/diagnostics", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/diagnostics/oracle.json", "implementation-invariant:parse-error-code"], "note": "Malformed source must produce the stable parse-error diagnostic."}, - {"fixture": "real-world/overpy-cake", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:real-world/overpy-cake/oracle.json", "provenance:real-world/overpy-cake/fixture.json"], "note": "Pinned OverPy example resolves through the native OPY semantic model."}, - {"fixture": "real-world/overpy-pixelart", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:real-world/overpy-pixelart/oracle.json", "provenance:real-world/overpy-pixelart/fixture.json"], "note": "Pinned OverPy example resolves; emitted rule shape is not the contract."}, - {"fixture": "real-world/overpy-santa", "nativeStatus": "success", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-santa/oracle.json", "regression:real-world/overpy-santa/fixture.json"], "note": "The full real-world project now compiles through the final #141 grammar and #144 builtin/member/enum residuals; canonical WIR equivalence remains a separate compiler gap."}, - {"fixture": "real-world/overpy-cronch", "nativeStatus": "success", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-cronch/oracle.json", "regression:real-world/overpy-cronch/fixture.json"], "note": "The final #141 grammar and #144 builtin/member/enum residuals now resolve the full project; canonical WIR equivalence remains a separate explicit compiler gap."}, - {"fixture": "real-world/overpy-broken-weapons", "nativeStatus": "success", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-broken-weapons/oracle.json", "regression:real-world/overpy-broken-weapons/fixture.json"], "note": "The final #141 return grammar and #144 builtin/member/enum residuals now resolve the full project; canonical WIR equivalence remains a separate explicit compiler gap."}, - {"fixture": "real-world/overpy-client-to-server", "nativeStatus": "success", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-client-to-server/oracle.json", "regression:real-world/overpy-client-to-server/fixture.json"], "note": "The final #141 grammar and #144 builtin/member/enum residuals now resolve the full project; canonical WIR equivalence remains a separate explicit compiler gap."}, - {"fixture": "real-world/overpy-crosshair", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:real-world/overpy-crosshair/oracle.json", "regression:real-world/overpy-crosshair/fixture.json"], "note": "The frontend resolves the full project through the audited HUD builtin surface; compiler-only lowering retains the separate stringModifier integration boundary."}, - {"fixture": "real-world/overpy-inputhud", "nativeStatus": "success", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-inputhud/oracle.json", "regression:real-world/overpy-inputhud/fixture.json"], "note": "The full real-world project now compiles through the final #141 grammar and #144 builtin/member/enum residuals; canonical WIR equivalence remains a separate compiler gap."}, - {"fixture": "real-world/overpy-parabola", "nativeStatus": "success", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/overpy-parabola/oracle.json", "regression:real-world/overpy-parabola/fixture.json"], "note": "The full real-world project now compiles through the final #141 grammar and #144 builtin/member/enum residuals; canonical WIR equivalence remains a separate compiler gap."}, - {"fixture": "real-world/overpy-meipocalypse", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:real-world/overpy-meipocalypse/oracle.json", "provenance:real-world/overpy-meipocalypse/fixture.json"], "note": "The reference rejects the missing script hook; native rejection is retained with a different earlier diagnostic."}, - {"fixture": "real-world/overpy-zencopter", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:real-world/overpy-zencopter/oracle.json", "provenance:real-world/overpy-zencopter/fixture.json"], "note": "Both sides reject the pinned upstream example; native diagnostic wording is not used as semantic evidence."}, - {"fixture": "real-world/ow1-emulator", "nativeStatus": "success", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:real-world/ow1-emulator/oracle.json", "provenance:real-world/ow1-emulator/fixture.json", "compiler-gate:unsupported-integration-surface"], "note": "The native frontend now resolves the complete include closure, while the pinned oracle snapshot still records the pre-fix parser failure; the compiler gate now passes the hero_roster indexed append/remove frontier and reaches the next existing unknown Hero filter boundary in heroes/mei/blaster."}, - {"fixture": "real-world/6v6-adjustments", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:real-world/6v6-adjustments/oracle.json", "provenance:real-world/6v6-adjustments/fixture.json", "implementation-invariant:source-frontier"], "note": "The full project reaches the pinned semantic unknown-member frontier after the demonstrated source constructs are parsed and lowered."}, - {"fixture": "synthetic/compiler-vertical-slice", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/compiler-vertical-slice/oracle.json", "implementation-invariant:opy-rs::compiler-vertical-slice"], "note": "The OPY source implementation resolves the source fixture; the compiler test independently lowers it through canonical WIR validation and deterministic workshop-rs emission."}, - {"fixture": "synthetic/compiler-structure", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/compiler-structure/oracle.json", "implementation-invariant:opy-rs::compiler-structural-lowering"], "note": "The pinned oracle records subroutine source identity, deterministic explicit/implicit variable allocation, and player event filters; the compiler test independently asserts those structures in canonical WIR."}, - {"fixture": "synthetic/primitive-lowering", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/primitive-lowering/oracle.json", "implementation-invariant:opy-rs::compiler-primitive-lowering", "test:opy-rs::compiler-primitive-lowering-equivalence"], "note": "The pinned oracle records assignment and modification lowering (including **= and single-level indexed forms), value expressions, array indexing (including firstOf index-0 normalization), not-comparison negation, implicit default variables at fixed slots, and non-null variable initializers; #145 additionally covers null-default omission, float-zero initialization, and nested indexed writes. The compiler test compares native lowered WIR directly with the parsed oracle WIR and asserts structural equivalence."}, - {"fixture": "synthetic/wait-optimization", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/wait-optimization/oracle.json", "provenance:synthetic/wait-optimization/fixture.json", "test:opy-rs::compiler::integration_tests::wait_lowering::optimized_wait_forms_match_the_pinned_oracle"], "note": "Issue #282: pinned OverPy 9.7.10 lowers wait() and sub-default durations to the canonical zero form under optimizeForSize, maps one-second duration to the canonical one form, and preserves the explicit wait behavior argument; native WIR is directly equivalent."}, - {"fixture": "synthetic/dictionary-lookup", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/dictionary-lookup/oracle.json", "implementation-invariant:opy-rs::compiler-literal-dict-lookup"], "note": "The source implementation and pinned oracle compile literal dict hits and misses; native lowering folds the selected key to its value or Null. Dictionary-indexed assignment targets remain a separate canonical-WIR boundary."}, - {"fixture": "synthetic/collection-mutation-328", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/collection-mutation-328/oracle.json", "provenance:synthetic/collection-mutation-328/fixture.json", "test:opy-rs::compiler::integration_tests::indexed_assignments::collection_deletion_matches_the_pinned_oracle"], "note": "Issue #328 collection deletion covers one-, two-, three-, and four-index global and player-variable targets through canonical indexed mutation lowering."}, - {"fixture": "synthetic/collection-mutation-328-invalid", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/collection-mutation-328-invalid/oracle.json", "provenance:synthetic/collection-mutation-328-invalid/fixture.json", "test:opy-rs::compiler::integration_tests::semantic_diagnostics::semantic_boundary_errors_are_reported_before_lowering"], "note": "Issue #328 pinned OverPy 9.7.10 accepts four source indexes but rejects a fifth; native preserves the five-index semantic boundary."}, - {"fixture": "synthetic/collection-mutation-328-random-invalid", "nativeStatus": "success", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:synthetic/collection-mutation-328-random-invalid/oracle.json", "provenance:synthetic/collection-mutation-328-random-invalid/fixture.json", "compiler-gate:unsupported-integration-surface", "test:opy-rs::compiler::integration_tests::indexed_assignments::nested_random_delete_reaches_the_pinned_boundary"], "note": "The source HIR resolves the target, while the compiler preserves OverPy 9.7.10's rejection of a random outer index in a three-index delete at the lowering boundary."}, - {"fixture": "synthetic/collection-mutation-328-random-player-invalid", "nativeStatus": "success", "classification": "known-gap", "ruleNames": false, "evidence": ["oracle:synthetic/collection-mutation-328-random-player-invalid/oracle.json", "provenance:synthetic/collection-mutation-328-random-player-invalid/fixture.json", "compiler-gate:unsupported-integration-surface", "test:opy-rs::compiler::integration_tests::indexed_assignments::nested_random_player_receiver_delete_reaches_the_pinned_boundary"], "note": "The source HIR resolves the target, while the compiler preserves OverPy 9.7.10's rejection when the player-variable receiver itself contains a random expression in a three-index delete."}, - {"fixture": "synthetic/indexed-assignment-nested", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/indexed-assignment-nested/oracle.json", "provenance:synthetic/indexed-assignment-nested/fixture.json", "test:opy-rs::compiler::integration_tests::indexed_assignments::nested_indexed_assignments_match_the_pinned_oracle"], "note": "Nested global indexed assignments, including compound modification and three-level replacement, are rebuilt through canonical array slices and match the pinned oracle WIR."}, - {"fixture": "synthetic/indexed-assignment-4d-invalid", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/indexed-assignment-4d-invalid/oracle.json", "provenance:synthetic/indexed-assignment-4d-invalid/fixture.json", "test:opy-rs::compiler::integration_tests::semantic_diagnostics::semantic_boundary_errors_are_reported_before_lowering"], "note": "The pinned OverPy implementation and native source semantics reject four-dimensional indexed assignments at the same semantic failure frontier."}, - {"fixture": "synthetic/postfix-assignment", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/postfix-assignment/oracle.json", "implementation-invariant:opy-rs::compiler-postfix-assignment", "test:opy-rs::compiler-postfix-assignment-equivalence"], "note": "The pinned oracle accepts postfix ++/-- assignments on global, player, and single-level indexed variables; native HIR resolves the forms and the compiler matches the canonical WIR."}, - {"fixture": "synthetic/postfix-prefix-invalid", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/postfix-prefix-invalid/oracle.json", "implementation-invariant:opy-rs::postfix-assignment-diagnostics"], "note": "The pinned oracle rejects prefix ++; native reports the stable source-attributed parse-error diagnostic, while prefix --x remains valid consecutive unary-minus syntax."}, - {"fixture": "synthetic/postfix-embedded-invalid", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/postfix-embedded-invalid/oracle.json", "implementation-invariant:opy-rs::postfix-assignment-diagnostics"], "note": "The pinned oracle independently rejects the embedded postfix form; native reports the stable source-attributed parse-error diagnostic."}, - {"fixture": "synthetic/range-player-variable", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/range-player-variable/oracle.json", "provenance:synthetic/range-player-variable/fixture.json", "test:opy-rs::compiler::integration_tests::range_bindings::player_variable_range_binder_matches_the_pinned_oracle"], "note": "The pinned hostPlayer.I range binder lowers to the existing canonical For Player Variable WIR contract and matches the oracle."}, - {"fixture": "synthetic/range-invalid-binder", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/range-invalid-binder/oracle.json", "provenance:synthetic/range-invalid-binder/fixture.json", "test:opy-rs::compiler::integration_tests::semantic_diagnostics::semantic_boundary_errors_are_reported_before_lowering"], "note": "The pinned oracle and native source semantics reject a non-variable range binder with a source-attributed diagnostic before canonical lowering."}, - {"fixture": "synthetic/member-values", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/member-values/oracle.json", "provenance:synthetic/member-values/fixture.json", "test:opy-rs::compiler::integration_tests::member_values::is_dummy_member_lowers_to_the_catalog_value_in_canonical_wir"], "note": "The catalog-backed eventPlayer.isDummy() member predicate resolves and lowers to the canonical Workshop value call."}, - {"fixture": "synthetic/hud-subheader", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/hud-subheader/oracle.json", "provenance:synthetic/hud-subheader/fixture.json", "test:opy-rs::compiler::integration_tests::hud::hud_subheader_matches_the_pinned_canonical_wir"], "note": "The shared hudSubheader action resolves and lowers to canonical Create HUD Text WIR with its argument and visibility contract."}, - {"fixture": "synthetic/hud-visibility", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/hud-visibility/oracle.json", "provenance:synthetic/hud-visibility/fixture.json", "test:opy-rs::compiler::integration_tests::hud_visibility::never_maps_to_visible_never_in_canonical_wir"], "note": "The OPY SpecVisibility.NEVER spelling resolves and lowers to the canonical VISIBLE_NEVER enum member."}, - {"fixture": "synthetic/control-flow-lowering", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/control-flow-lowering/oracle.json", "implementation-invariant:control-flow-lowering-hir"], "note": "The #47 control-flow slice resolves in the source implementation and its native lowered WIR is directly equivalent to the pinned oracle."}, - {"fixture": "synthetic/conditional-forward-gotos", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/conditional-forward-gotos/oracle.json", "provenance:synthetic/conditional-forward-gotos/fixture.json", "test:opy-rs::compiler::integration_tests::control_flow::conditional_forward_gotos_match_the_pinned_oracle"], "note": "Conditional forward gotos preserve pinned OverPy branch structure, full structured-action distances, and single condition evaluation."}, - {"fixture": "synthetic/control-flow-328", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/control-flow-328/oracle.json", "provenance:synthetic/control-flow-328/fixture.json", "test:opy-rs::compiler::integration_tests::control_flow::issue_328_control_flow_matches_the_pinned_oracle"], "note": "Issue #328 continue, do-while, dynamic loc+, and RULE_START forms resolve and match the pinned canonical Workshop boundary."}, - {"fixture": "synthetic/switch-break-unsupported", "nativeStatus": "success", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/switch-break-unsupported/oracle.json", "test:opy-rs::compiler::integration_tests::control_flow::nested_switch_break_matches_upstream_noop_elision"], "note": "#47 probe: nested conditional switch-break lowering preserves the upstream Else marker, and the top-level no-op switch is elided like the pinned oracle."}, - {"fixture": "synthetic/switch-order", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/switch-order/oracle.json", "implementation-invariant:authored-switch-order"], "note": "The #47 default-before-case probe preserves authored arm order and fallthrough in the source implementation, and the native lowered WIR is directly equivalent to the pinned oracle."}, - {"fixture": "synthetic/switch-structured-target", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/switch-structured-target/oracle.json", "implementation-invariant:structured-switch-source-order"], "note": "The #47 structured switch probe preserves nested if/while actions and authored case/default target order in the source implementation, and the native lowered WIR is directly equivalent to the pinned oracle."}, - {"fixture": "synthetic/switch-multiple-break", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/switch-multiple-break/oracle.json", "test:opy-rs::compiler::integration_tests::control_flow::multiple_switch_breaks_match_independent_semantic_oracle", "test:opy-rs::compiler::integration_tests::control_flow::pinned_overpy_switch_action_trace"], "note": "The source implementation preserves the multi-break source; the compiler lowers its direct case-local breaks through canonical Workshop control flow, and the pinned OverPy action trace independently checks case selection, break, default, and fallthrough behavior."}, - {"fixture": "synthetic/do-while-break", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:synthetic/do-while-break/oracle.json", "test:opy-rs::compiler::integration_tests::control_flow::do_while_break_shapes_match_the_pinned_oracle"], "note": "Direct, conditional, and nested do-while break shapes resolve and match the pinned Workshop through direct native-WIR comparison with the parsed oracle."}, - {"fixture": "synthetic/do-while-invalid", "nativeStatus": "failure", "classification": "match", "ruleNames": false, "evidence": ["oracle:synthetic/do-while-invalid/oracle.json", "test:opy-rs::compiler::integration_tests::control_flow::invalid_do_while_placement_is_source_attributed"], "note": "The source implementation reports the stable source-attributed do-while-placement diagnostic for a non-prefix do-while."}, - {"fixture": "census/workshop-feature-census", "nativeStatus": "success", "classification": "match", "ruleNames": true, "evidence": ["oracle:census/workshop-feature-census/oracle.json", "contract:workshop-rs#10-feature-census"], "note": "OPy source records opaque Workshop feature identities for the future workshop-rs lowering boundary."} - ] -} diff --git a/crates/opy-rs/tests/differential.rs b/crates/opy-rs/tests/differential.rs index d4608ea8..76b7f694 100644 --- a/crates/opy-rs/tests/differential.rs +++ b/crates/opy-rs/tests/differential.rs @@ -1,26 +1,24 @@ //! Native-vs-reference differential suite. //! -//! Runs the declared compatibility corpus +//! Runs the compatibility corpus //! (`crates/opy-rs/tests/fixtures/corpus/**/fixture.json`) through the native frontend and -//! compares the outcome against the recorded reference evidence +//! compares the outcome against the recorded reference result //! (`oracle.json`, produced by `tools/overpy/run_oracle.py` against the //! pinned OverPy 9.7.10 oracle). //! //! # What is compared //! -//! The recorded reference evidence is the oracle's **Workshop text output**, +//! The recorded reference result contains the oracle's **Workshop text output**, //! not a reference HIR (the wright adapter HIR fixtures do not exist in this //! repository). The parity contract is observable semantics, not byte -//! identity, so the suite compares at the boundary the evidence supports: +//! identity, so the suite compares the supported observable boundaries: //! //! * **Status parity** — the native frontend must resolve a fixture the //! oracle accepts, and must reject a fixture the oracle rejects (with a //! structured diagnostic). This is the primary, CI-enforced contract. -//! * **Expected-outcome table** — every fixture has an explicit expectation -//! (`resolve` or `expected-diagnostic`) with a documented rationale, so a -//! fixture that is legitimately unsupported is a *documented* entry, not a -//! silent failure; behavior that leaves the table is a `divergence` and -//! fails the suite (regressions break CI, mirroring the wright contract). +//! * **Expected outcome** — each fixture manifest declares the native outcome +//! and its relationship to the oracle; behavior outside that declaration is +//! a `divergence` and fails the suite. //! * **Structural self-check** (always runs) — a resolved program must pass //! Opy HIR v2 validation, must round-trip through the wire payload //! (`parse_value(serde_json::to_value(program))`), and its debug dump must @@ -39,14 +37,14 @@ //! //! * Span endpoints (`span` objects) are removed from the emitted native HIR //! JSON dumps (`target/opy-differential/.native.json`): -//! frontend-internal provenance that the reference evidence does not +//! frontend-internal source mapping that the reference result does not //! record. `protocol`/`generator` identities are contract fields and are //! kept verbatim. //! * Reference synthesized rules (`Initialize global variables`, //! `Initialize player variables`, `Subroutine …`) are dropped from the //! rule-name comparison because their emission is lowering-dependent. //! * Diagnostic wording is never compared; only status and the stable -//! diagnostic `code` (where the expectation table pins one). +//! diagnostic `code` (where the fixture manifest pins one). //! //! # Degradation //! @@ -65,18 +63,6 @@ //! reference status, and rule-name comparison. A reference-success/native-failure case is never //! classified as a match. //! -//! # Current corpus state -//! -//! All declared fixtures run (0 skips, 0 divergences): **29 resolve** and -//! **16 produce expected diagnostics** with pinned codes; 7 fixtures are -//! documented reference gaps (the oracle accepts a surface the native -//! frontend deliberately rejects). Settings key-existence/leaf-kind -//! validation and Workshop enum member/domain validation were removed from -//! the frontend core (ownership fix): the affected fixtures resolve -//! structurally with opaque Workshop identity, and the checks are -//! `lowering-dependent` (issue #8) — the expectation table reflects that -//! contract. - use std::collections::BTreeMap; use std::path::{Path, PathBuf}; @@ -91,513 +77,6 @@ fn fixtures_root() -> PathBuf { workspace_root().join("crates/opy-rs/tests/fixtures/corpus") } -fn differential_expectations() -> Value { - serde_json::from_str( - &std::fs::read_to_string( - workspace_root().join("crates/opy-rs/tests/differential-expectations.json"), - ) - .expect("differential-expectations.json must be readable"), - ) - .expect("differential-expectations.json must parse") -} - -fn expectation_for<'a>(expectations: &'a Value, id: &str) -> &'a Value { - expectations["cases"] - .as_array() - .expect("differential expectations must contain cases") - .iter() - .find(|case| case["fixture"].as_str() == Some(id)) - .unwrap_or_else(|| panic!("fixture '{id}' is missing from differential expectations")) -} - -/// What the native frontend is expected to do for a fixture on this branch. -#[derive(Clone, Copy, PartialEq, Eq, Debug)] -enum Expect { - /// The native frontend must resolve the fixture to an Opy HIR program. - Resolve, - /// The native frontend must reject the fixture with a structured - /// diagnostic; `code` pins the stable code when known. - Diagnostic { code: Option<&'static str> }, -} - -#[derive(Clone, Copy)] -struct Case { - expect: Expect, - /// Compare authored rule names against the oracle Workshop text. - rule_names: bool, - /// Documented rationale; also the divergence entry when the table is - /// deliberately not matching the oracle status. - note: &'static str, -} - -fn case(expect: Expect, rule_names: bool, note: &'static str) -> Case { - Case { - expect, - rule_names, - note, - } -} - -/// The declared corpus expectation table. Every fixture in -/// `crates/opy-rs/tests/fixtures/corpus` must appear here; unknown fixtures fail the suite -/// so new corpus entries are deliberate. -fn declared_corpus() -> BTreeMap<&'static str, Case> { - let mut cases = BTreeMap::new(); - let resolve = |cases: &mut BTreeMap<&'static str, Case>, id, rule_names, note| { - cases.insert(id, case(Expect::Resolve, rule_names, note)); - }; - let diagnostic = |cases: &mut BTreeMap<&'static str, Case>, id, code, note| { - cases.insert(id, case(Expect::Diagnostic { code }, false, note)); - }; - - // Synthetic fixtures (WrightKit-authored, AGPL-3.0-or-later). - resolve( - &mut cases, - "synthetic/basic-rule", - true, - "minimal rule; oracle status success (parity).", - ); - resolve( - &mut cases, - "synthetic/control-flow", - true, - "if/elif/else, for-in-range, while, pass; oracle status success (parity).", - ); - resolve( - &mut cases, - "synthetic/syntax-surface", - true, - "Issue #28 pure OPY syntax: switch, do-while, hex, membership, dict indexing, comprehensions, lambda arguments, and string modifiers.", - ); - resolve( - &mut cases, - "synthetic/string-modifiers", - true, - "Issue #28 inventory-backed string modifiers; translation-dependent l/t are syntax-carried outside the fixture.", - ); - resolve( - &mut cases, - "synthetic/switch-break", - true, - "Issue #33 switch arms preserve source-order fallthrough and explicit break statements validate nested switch/loop context.", - ); - resolve( - &mut cases, - "synthetic/strings-and-lambda", - true, - "Issue #33 f-string interpolation preserves source-spanned expressions and the approved sorted lambda argument slot.", - ); - resolve( - &mut cases, - "synthetic/compiler-vertical-slice", - true, - "Issue #35 OPY-to-Workshop integration fixture; the frontend resolves the source and the internal compiler module independently validates the canonical WIR slice.", - ); - resolve( - &mut cases, - "synthetic/compiler-structure", - false, - "Issue #40 oracle-backed structural probe; the frontend resolves the source while the internal compiler module independently checks canonical WIR identity, allocation, and event filters.", - ); - resolve( - &mut cases, - "synthetic/primitive-lowering", - false, - "Issue #46 oracle-backed non-control-flow primitives probe; the frontend resolves the source while the compiler test suite constrains native lowering against the pinned oracle through the canonical workshop-rs parser and structural equivalence.", - ); - resolve( - &mut cases, - "synthetic/dictionary-lookup", - false, - "Issue #46 negative primitive-lowering probe; the frontend and the pinned oracle accept the dict-indexed assignment while the compiler rejects it with the stable source-attributed unsupported-integration-surface diagnostic.", - ); - resolve( - &mut cases, - "synthetic/postfix-assignment", - false, - "Issue #59 postfix ++/-- assignment probe; global, player, and single-level indexed targets resolve and are constrained by compiler oracle tests.", - ); - resolve( - &mut cases, - "synthetic/indexed-assignment-nested", - false, - "Issue #60 nested indexed assignment probe; global targets resolve and are constrained by the pinned canonical-WIR oracle.", - ); - resolve( - &mut cases, - "synthetic/collection-mutation-328", - true, - "Issue #328 pinned collection-deletion probe; one- through four-index global and player-variable targets resolve through canonical indexed mutation lowering.", - ); - diagnostic( - &mut cases, - "synthetic/collection-mutation-328-invalid", - Some("four-dimensional-delete"), - "Issue #328 pinned five-index deletion rejection; the native semantic boundary matches the pinned OverPy rejection frontier.", - ); - resolve( - &mut cases, - "synthetic/collection-mutation-328-random-invalid", - false, - "Issue #328 pins the compiler-level rejection of a random outer index in a three-index delete; the source HIR remains structurally resolvable and the compiler expectation records the lowering boundary.", - ); - resolve( - &mut cases, - "synthetic/collection-mutation-328-random-player-invalid", - false, - "Issue #328 pins the compiler-level rejection of a random player receiver in a three-index delete; the source HIR remains structurally resolvable and the compiler expectation records the lowering boundary.", - ); - resolve( - &mut cases, - "synthetic/wait-optimization", - false, - "Issue #282 minimized wait probe; optimizeForSize lowers omitted and sub-default durations to the pinned OverPy boolean forms while preserving explicit duration and reevaluation behavior.", - ); - diagnostic( - &mut cases, - "synthetic/indexed-assignment-4d-invalid", - Some("four-dimensional-assignment"), - "Issue #60 four-dimensional negative probe; source semantics reject the assignment at a stable, source-attributed diagnostic frontier before canonical lowering.", - ); - diagnostic( - &mut cases, - "synthetic/postfix-prefix-invalid", - Some("parse-error"), - "Issue #59 prefix ++ remains a stable source-attributed parse error; prefix --x remains valid consecutive unary-minus syntax.", - ); - diagnostic( - &mut cases, - "synthetic/postfix-embedded-invalid", - Some("parse-error"), - "Issue #59 embedded postfix form remains a stable source-attributed parse error with independent pinned oracle evidence.", - ); - resolve( - &mut cases, - "synthetic/range-player-variable", - false, - "Issue #65 player-variable range binder resolves in HIR; canonical For Player Variable lowering is constrained by the dedicated compiler test.", - ); - diagnostic( - &mut cases, - "synthetic/range-invalid-binder", - Some("invalid-range-binder"), - "Issue #65 non-variable range binder is rejected by source semantics with a stable source-attributed diagnostic before canonical lowering.", - ); - resolve( - &mut cases, - "synthetic/array-unique", - true, - "Issue #322 Array.unique() resolves as an OPY member; canonical filtered-array lowering is constrained by the dedicated compiler test.", - ); - resolve( - &mut cases, - "synthetic/member-values", - true, - "Issue #113 catalog-backed eventPlayer.isDummy() member predicate; canonical WIR lowering is constrained by the dedicated compiler test.", - ); - resolve( - &mut cases, - "synthetic/hud-subheader", - true, - "Issue #114 shared hudSubheader action; canonical WIR lowering is constrained by the dedicated compiler test.", - ); - resolve( - &mut cases, - "synthetic/member-angle", - true, - "Issue #130 catalog-backed eventPlayer.getHorizontalFacingAngle() member value; canonical WIR lowering is constrained by the dedicated compiler test.", - ); - resolve( - &mut cases, - "synthetic/hud-visibility", - true, - "Issue #131 maps SpecVisibility.NEVER to the canonical VISIBLE_NEVER member; canonical WIR lowering is constrained by the dedicated compiler test.", - ); - resolve( - &mut cases, - "synthetic/control-flow-lowering", - false, - "Issue #47 oracle-backed control-flow lowering probe; the frontend resolves the source while the compiler independently constrains canonical WIR against the pinned oracle.", - ); - resolve( - &mut cases, - "synthetic/conditional-forward-gotos", - true, - "Issue #319 oracle-backed forward-goto probe; branch structure, action distances, and condition evaluation remain aligned with the pinned oracle.", - ); - resolve( - &mut cases, - "synthetic/control-flow-328", - true, - "Issue #328 pinned continue, do-while, dynamic loc+, and RULE_START probe; accepted control-flow forms resolve through canonical WIR lowering.", - ); - resolve( - &mut cases, - "synthetic/switch-break-unsupported", - true, - "Issue #47 probe; nested conditional switch-break lowering preserves the upstream Else marker, and a top-level switch with no observable actions is elided like OverPy.", - ); - resolve( - &mut cases, - "synthetic/switch-order", - false, - "Issue #47 source-order switch probe; default-before-case fallthrough remains represented in ordered HIR arms.", - ); - resolve( - &mut cases, - "synthetic/switch-structured-target", - false, - "Issue #47 structured switch-target probe; nested canonical control-flow widths preserve later case/default targets.", - ); - resolve( - &mut cases, - "synthetic/switch-multiple-break", - false, - "Issue #47 multi-break probe; the frontend preserves all authored arms and breaks while the compiler lowers them through canonical nested switch-exit WIR.", - ); - resolve( - &mut cases, - "synthetic/do-while-break", - false, - "Issue #47 do-while probe; direct, conditional, and nested break shapes resolve in the frontend and are constrained by compiler oracle tests.", - ); - diagnostic( - &mut cases, - "synthetic/do-while-invalid", - Some("do-while-placement"), - "Issue #47 invalid do-while placement remains a stable source-attributed frontend diagnostic.", - ); - diagnostic( - &mut cases, - "synthetic/lambda-invalid", - Some("lambda-context"), - "Issue #33 standalone lambda use remains rejected outside a signature-approved argument position.", - ); - diagnostic( - &mut cases, - "synthetic/syntax-invalid", - Some("parse-error"), - "Issue #28 malformed do-while and dictionary syntax remains a structured parse failure.", - ); - resolve( - &mut cases, - "synthetic/declarations-numbers", - true, - "numeric literals and variable-index declarations; oracle status success.", - ); - resolve( - &mut cases, - "synthetic/declarations-rules", - true, - "globalvar/playervar/subroutine/def/enum and rule headers; oracle status success.", - ); - resolve( - &mut cases, - "synthetic/expressions-values", - true, - "expressions, arrays, strings, vectors, calls, .format; oracle status success.", - ); - resolve( - &mut cases, - "synthetic/preprocessing", - false, - "include + object/function-like defines; oracle status success.", - ); - resolve( - &mut cases, - "synthetic/directives", - true, - "advanced directive state and source annotations; oracle status success.", - ); - diagnostic( - &mut cases, - "synthetic/duplicate-rule-diagnostic", - Some("duplicate-rule-name"), - "ordinary-rule @Name reaches the pinned duplicate-rule-name semantic frontier; oracle status failure.", - ); - resolve( - &mut cases, - "synthetic/project-main-file", - true, - "mainFile entry-point redirect and child-include scope; oracle status success.", - ); - resolve( - &mut cases, - "synthetic/project-entry", - true, - "included-file mainFile scope and root entry-point preservation; oracle status success.", - ); - resolve( - &mut cases, - "synthetic/legacy-entry-include", - true, - "entry include paths continue from the latest resolved file base, matching the pinned OverPy project topology.", - ); - resolve( - &mut cases, - "synthetic/directives-scoped", - false, - "pinned positive probe for global rulePrefixTemplate, include prefix restoration, AST macro/enum redeclaration, and translation normalization.", - ); - diagnostic( - &mut cases, - "synthetic/translations-invalid", - Some("translations-invalid"), - "pinned negative probe for a language code outside the exact translation set.", - ); - resolve( - &mut cases, - "synthetic/include-scope", - false, - "nested include optimization directives are retained as observable scoped state; strict-sensitive lowering is covered by the compiler compatibility contract.", - ); - resolve( - &mut cases, - "synthetic/optimize-strict", - true, - "pinned strict optimizer probe retains expressions whose type-conversion semantics must not be rewritten.", - ); - resolve( - &mut cases, - "synthetic/settings", - true, - "top-of-file settings block parsed into the typed HIR payload; validation is structural only (group shape, span validity, non-empty key names) — key-existence/leaf-kind checks were removed from the core and are lowering-dependent (#8); oracle status success.", - ); - resolve( - &mut cases, - "synthetic/receiver-calls", - true, - "receiver/member call forms; oracle status success.", - ); - resolve( - &mut cases, - "synthetic/receiver-playervar", - true, - "bare variable member expression is preserved as an OPY HIR member node; canonical Workshop member validation remains lowering-dependent.", - ); - resolve( - &mut cases, - "synthetic/chase-enums", - true, - "ChaseTimeReeval/ChaseRateReeval member accesses resolve as opaque Workshop enum identities (member-existence/domain validation was removed from the core and is lowering-dependent, #8); oracle status success.", - ); - resolve( - &mut cases, - "synthetic/chase-condition-agentlab", - true, - "chaseOverTime in rule conditions (agent-lab regression); oracle status success.", - ); - resolve( - &mut cases, - "synthetic/chase-keywords", - true, - "named/keyword arguments and chase/ChaseReeval forms; the contextual member rewrites to the keyword-selected domain without membership checks (lowering-dependent, #8); oracle status success.", - ); - resolve( - &mut cases, - "synthetic/for-range-agentlab", - true, - "for with implicit default-variable binder; oracle status success.", - ); - diagnostic( - &mut cases, - "synthetic/diagnostics", - Some("parse-error"), - "expected-failure fixture: the native frontend rejects missing-colon with parse-error; oracle status failure (parity).", - ); - resolve( - &mut cases, - "census/workshop-feature-census", - true, - "OPy-side Workshop feature census boundary; canonical feature identities remain owned by workshop-rs#10.", - ); - - // Real-world fixtures derived from upstream OverPy examples (GPL-3.0-only, - // provenance-recorded evidence; oracle status success). - resolve( - &mut cases, - "real-world/overpy-cake", - true, - "cake example; oracle status success (parity).", - ); - resolve( - &mut cases, - "real-world/overpy-pixelart", - false, - "pixelart example; oracle status success.", - ); - resolve( - &mut cases, - "real-world/overpy-santa", - false, - "the final #141 grammar and #144 builtin/member/enum residuals now resolve the full project; canonical WIR equivalence remains a separate explicit compiler gap.", - ); - resolve( - &mut cases, - "real-world/overpy-cronch", - false, - "the final #141 grammar and #144 builtin/member/enum residuals now resolve the full project; canonical WIR equivalence remains a separate explicit compiler gap.", - ); - resolve( - &mut cases, - "real-world/overpy-broken-weapons", - false, - "the final #141 return grammar and #144 builtin/member/enum residuals now resolve the full project; canonical WIR equivalence remains a separate explicit compiler gap.", - ); - resolve( - &mut cases, - "real-world/overpy-client-to-server", - false, - "the final #141 grammar and #144 builtin/member/enum residuals now resolve the full project; canonical WIR equivalence remains a separate explicit compiler gap.", - ); - resolve( - &mut cases, - "real-world/overpy-crosshair", - false, - "the frontend resolves the full project through the audited HUD builtin surface; compiler-only lowering retains the separate stringModifier integration boundary.", - ); - resolve( - &mut cases, - "real-world/overpy-inputhud", - false, - "the final #141 grammar and #144 builtin/member/enum residuals now resolve the full project; canonical WIR equivalence remains a separate explicit compiler gap.", - ); - resolve( - &mut cases, - "real-world/overpy-parabola", - false, - "the final #141 grammar and #144 builtin/member/enum residuals now resolve the full project; canonical WIR equivalence remains a separate explicit compiler gap.", - ); - - // Real-world failure fixtures (reference rejects; recorded diagnostics). - diagnostic( - &mut cases, - "real-world/overpy-meipocalypse", - Some("lex-error"), - "reference fails with ENOENT on the __script__ macros (JS files not ported); the native frontend rejects earlier on the dict-literal surface (baseline category 1b, explicit rejection). Gap: rejection reason differs (documented); the script-macro defines are lexed as directives but never reached.", - ); - diagnostic( - &mut cases, - "real-world/overpy-zencopter", - Some("lex-error"), - "reference rejects the example ('Invalid content before string: 'arena'', upstream example bug); the native frontend rejects the triple-quoted-string surface (baseline category 1b, declared rejection, unterminated-string lex error). Gap: rejection reasons differ (documented).", - ); - resolve( - &mut cases, - "real-world/ow1-emulator", - false, - "the pinned oracle snapshot still records the pre-fix parser failure, while the native frontend resolves the complete include closure; the compiler gate now passes the hero_roster indexed append/remove frontier and reaches the next existing unknown Hero filter boundary in heroes/mei/blaster.", - ); - diagnostic( - &mut cases, - "real-world/6v6-adjustments", - Some("unknown-member"), - "the native frontend reaches the pinned unknown-member semantic frontier after resolving the demonstrated source constructs.", - ); - - cases -} - /// Recursively collect `fixture.json` paths under `root`, sorted. fn discover_fixtures(root: &Path) -> Vec { let mut out = Vec::new(); @@ -619,14 +98,25 @@ fn discover_fixtures(root: &Path) -> Vec { out } -/// Load a fixture manifest, returning `(id, source_name, expected_status)`. -fn load_fixture(path: &Path) -> (String, String, String) { +/// Load a fixture manifest, returning its source-test expectation. +fn load_fixture(path: &Path) -> (String, String, String, String, bool, Option) { let manifest: Value = serde_json::from_str(&std::fs::read_to_string(path).unwrap()) .unwrap_or_else(|error| panic!("cannot parse {}: {error}", path.display())); let id = manifest["id"].as_str().unwrap().to_string(); let source = manifest["source"].as_str().unwrap().to_string(); - let expected = manifest["expectedStatus"].as_str().unwrap().to_string(); - (id, source, expected) + let source_test = &manifest["tests"]["source"]; + let native_status = source_test["nativeStatus"].as_str().unwrap().to_string(); + let relationship = source_test["relationship"].as_str().unwrap().to_string(); + let rule_names = source_test["ruleNames"].as_bool().unwrap(); + let diagnostic_code = source_test["diagnosticCode"].as_str().map(str::to_string); + ( + id, + source, + native_status, + relationship, + rule_names, + diagnostic_code, + ) } /// Collect authored rule names from the native HIR in program order. @@ -721,9 +211,7 @@ fn run_native( } #[test] -fn native_and_reference_agree_on_the_declared_corpus() { - let corpus = declared_corpus(); - let expectations = differential_expectations(); +fn native_and_reference_agree_on_the_corpus() { let mut reference_identity = Value::Null; let mut fixtures = BTreeMap::::new(); @@ -742,39 +230,15 @@ fn native_and_reference_agree_on_the_declared_corpus() { "skipped": 0 }); - let mut seen: Vec = Vec::new(); for manifest_path in discover_fixtures(&fixtures_root()) { - let (id, source_name, expected_status) = load_fixture(&manifest_path); - seen.push(id.clone()); - let case = *corpus.get(id.as_str()).unwrap_or_else(|| { - panic!( - "fixture '{id}' is not declared in the differential expectation table; \ - add an explicit resolve/diagnostic entry with a note" - ) - }); - let expectation = expectation_for(&expectations, &id); - let expected_native_status = expectation["nativeStatus"] - .as_str() - .unwrap_or_else(|| panic!("{id}: nativeStatus is required")); - let expected_classification = expectation["classification"] - .as_str() - .unwrap_or_else(|| panic!("{id}: classification is required")); - let expected_evidence = expectation["evidence"] - .as_array() - .unwrap_or_else(|| panic!("{id}: evidence is required")); - assert!( - !expected_evidence.is_empty(), - "{id}: differential expectation evidence cannot be empty" - ); - assert_eq!( + let ( + id, + source_name, expected_native_status, - if matches!(case.expect, Expect::Resolve) { - "success" - } else { - "failure" - }, - "{id}: differential expectation disagrees with native expectation table" - ); + expected_relationship, + compare_rule_names, + expected_diagnostic_code, + ) = load_fixture(&manifest_path); let fixture_dir = manifest_path.parent().unwrap().to_path_buf(); let source_path = fixture_dir.join(&source_name); let source = std::fs::read_to_string(&source_path) @@ -808,12 +272,11 @@ fn native_and_reference_agree_on_the_declared_corpus() { None }; let reference_entry = json!({ - "expectedStatus": expected_status, "snapshot": if snapshot_present { "present" } else { "absent" }, - "compileStatus": reference_status, + "compileStatus": reference_status.as_deref(), }); - // Status determination against the expectation table. + // Status determination against the fixture manifest. let expect_resolve = expected_native_status == "success"; let status = if native_ok == expect_resolve { if expect_resolve { @@ -833,14 +296,8 @@ fn native_and_reference_agree_on_the_declared_corpus() { // Pinned diagnostic codes must match exactly. let mut detail = Vec::new(); - if let ( - Expect::Diagnostic { - code: Some(expected_code), - }, - Err(error), - ) = (case.expect, &native) - { - if error.code != expected_code { + if let (Some(expected_code), Err(error)) = (&expected_diagnostic_code, &native) { + if error.code != expected_code.as_str() { detail.push(format!( "expected diagnostic code '{expected_code}', got '{}'", error.code @@ -848,9 +305,8 @@ fn native_and_reference_agree_on_the_declared_corpus() { } } - // Reference status parity: informational when the expectation table - // deliberately diverges from the oracle (the entry is then a - // documented `referenceGap`; hard divergences fail the suite). + // Reference status parity: informational when the manifest deliberately + // declares a known gap between native and reference behavior. let reference_gap = match &reference_status { Some(reference_status) => (*reference_status == "success") != native_ok, None => false, @@ -863,33 +319,23 @@ fn native_and_reference_agree_on_the_declared_corpus() { )); } - // Oracle snapshot consistency: the recorded compile status must match - // the fixture manifest's expectedStatus. - if let Some(compile_status) = &reference_status { - if compile_status != &expected_status { - detail.push(format!( - "oracle.json compile.status '{compile_status}' disagrees with fixture.json expectedStatus '{expected_status}'" - )); - } - } - - let relationship_holds = match expected_classification { + let relationship_holds = match expected_relationship.as_str() { "match" => !reference_gap, "known-gap" | "unsupported" => reference_gap, - other => panic!("{id}: unsupported expectation classification '{other}'"), + other => panic!("{id}: unsupported relationship '{other}'"), }; let classification = if skipped { "inconclusive" } else if status == "divergence" || !relationship_holds { "unexpected-divergence" } else if reference_gap { - expected_classification + expected_relationship.as_str() } else { "match" }; // Rule-name parity (informational, opt-in per fixture). - let rule_names_entry = if case.rule_names && snapshot_present { + let rule_names_entry = if compare_rule_names && snapshot_present { match &native { Ok(program) => { let oracle: Value = @@ -917,7 +363,6 @@ fn native_and_reference_agree_on_the_declared_corpus() { let entry = json!({ "status": status, "expect": if expect_resolve { "resolve" } else { "diagnostic" }, - "note": case.note, "native": native_entry, "reference": reference_entry, "ruleNames": rule_names_entry, @@ -925,8 +370,7 @@ fn native_and_reference_agree_on_the_declared_corpus() { "referenceGap": reference_gap, "skip": skipped, "classification": classification, - "expectedClassification": expected_classification, - "evidence": expected_evidence, + "expectedRelationship": expected_relationship, }); let code = entry["native"].get("code").and_then(Value::as_str); let label = if skipped { @@ -955,7 +399,6 @@ fn native_and_reference_agree_on_the_declared_corpus() { "expect": if expect_resolve { "resolve" } else { "diagnostic" }, "native": entry["native"], "reference": entry["reference"], - "note": case.note, "detail": detail, })); } @@ -964,7 +407,6 @@ fn native_and_reference_agree_on_the_declared_corpus() { "fixture": id, "native": entry["native"], "reference": entry["reference"], - "note": case.note, "detail": detail, })); } @@ -996,31 +438,9 @@ fn native_and_reference_agree_on_the_declared_corpus() { fixtures.insert(id.clone(), entry); } - // Every declared corpus entry must exist on disk. - let missing: Vec<&str> = corpus - .keys() - .filter(|id| !seen.iter().any(|s| s == *id)) - .copied() - .collect(); - assert!( - missing.is_empty(), - "declared corpus entries missing from crates/opy-rs/tests/fixtures/corpus: {missing:?}" - ); - let extra_expectations: Vec<&str> = expectations["cases"] - .as_array() - .expect("differential expectations must contain cases") - .iter() - .filter_map(|case| case["fixture"].as_str()) - .filter(|id| !seen.iter().any(|seen_id| seen_id == id)) - .collect(); - assert!( - extra_expectations.is_empty(), - "differential expectations reference missing fixtures: {extra_expectations:?}" - ); - let report = json!({ "schemaVersion": 1, - "artifact": "opy-rs native-vs-reference differential report (issue #25)", + "artifact": "opy-rs native-vs-reference differential report", "generatedBy": "crates/opy-rs/tests/differential.rs", "frontend": { "name": LANGUAGE_NAME, "version": LANGUAGE_VERSION }, "reference": reference_identity, @@ -1058,7 +478,7 @@ fn native_and_reference_agree_on_the_declared_corpus() { entry["fixture"], entry["expect"], entry["detail"].as_array().map_or_else( - || entry["note"].to_string(), + || "no detail".to_string(), |list| { list.iter() .map(Value::to_string) diff --git a/crates/opy-rs/tests/fixtures/corpus/README.md b/crates/opy-rs/tests/fixtures/corpus/README.md index 95d72a50..ce7b0e82 100644 --- a/crates/opy-rs/tests/fixtures/corpus/README.md +++ b/crates/opy-rs/tests/fixtures/corpus/README.md @@ -1,155 +1,56 @@ -# Compatibility Fixture Corpus — Provenance +# OPY compatibility fixtures -This directory is the opy-rs compatibility corpus: OPY sources with their -pinned-oracle snapshots (`oracle.json`), ported from the WrightKit project's -evidence base and re-verified against the pinned OverPy 9.7.10 oracle (see -[`docs/compatibility/upstream-references.md`](../../../../../docs/compatibility/upstream-references.md) -for the dated verification record). - -Corpus policy: every fixture records provenance in its `fixture.json` -(`kind`, `origin`, `license`, `redistributable`, and — for imported -fixtures — `sourceCommit`, `sourceUrl`, `licenseUrl`, `modifications`). -No fixture is committed without a clear provenance record. The pinned -reference identity and the clean-room policy are in -[`docs/compatibility/upstream-references.md`](../../../../../docs/compatibility/upstream-references.md). -These files are oracle evidence, not core inputs: the core never imports -them, and they are never bundled into core builds or release artifacts. +This directory contains OPY inputs and pinned OverPy results used by the +source and compiler differential tests. Each fixture is ordinary test data: +the native implementation runs against the input, the pinned OverPy 9.7.10 +result provides the reference behavior, and the runners assert the declared +relationship. ## Layout ```text corpus/// - fixture.json # metadata and expected compile status - source.opy # input (or the source path named by fixture.json; multi-file - # fixtures name their main file via fixture.json "source") - oracle.json # normalized pinned-oracle snapshot (compile status, exit code, - # diagnostics, normalized Workshop text, output hash) + fixture.json # input, optional attribution, and test expectations + source.opy # input, or the source path named by fixture.json + oracle.json # pinned reference result snapshot ``` -## Synthetic fixtures (WrightKit-authored) - -`fixtures/synthetic/` — fixtures authored for the WrightKit compatibility -corpus (AGPL-3.0-or-later, `kind: original`): an initial set ported unchanged -from the wright repository corpus, extended by fixtures added in this -repository: - -| Fixture | Covers | -| --- | --- | -| `basic-rule` | minimal rule with `@Event global`, `disableInspector()` | -| `control-flow` | `if`/`elif`/`else`, `for … in range`, `while`, `pass` | -| `declarations-rules` | `globalvar`/`playervar`/`subroutine`/`def`/`enum`, rule headers | -| `declarations-numbers` | numeric literal forms and variable-index declarations | -| `expressions-values` | expressions, arrays, strings, vectors, calls, `.format` | -| `preprocessing` | `#!include` (with `shared.opy`), `#!define` object/function-like | -| `directives-scoped` | Pinned positive probe for rule-prefix templates, include prefix restoration, macro/enum redeclaration, and normalized translations | -| `translations-invalid` | Pinned negative probe for a translation code outside the oracle's exact set | -| `include-scope` | Pinned nested-include probe for observable optimization state transitions | -| `optimize-strict` | #288 pinned strict-optimizer probe for expressions whose type-conversion semantics must be preserved | -| `diagnostics` | expected-failure fixture with a syntax diagnostic | -| `settings` | top-of-file `settings { … }` JSONC block | -| `receiver-calls` | receiver/member call forms (derived from the real-world overpy-meipocalypse corpus; see its `fixture.json` provenance note) | -| `chase-enums` | `ChaseTimeReeval`/`ChaseRateReeval` enum domains | -| `chase-condition-agentlab` | `chaseOverTime(...)` in rule conditions (agent-lab regression) | -| `chase-keywords` | named/keyword arguments and the `chase`/`ChaseReeval` contextual forms | -| `for-range-agentlab` | `for` with implicit default-variable binder (agent-lab regression, `kind: derived`) | -| `syntax-surface`, `syntax-invalid`, `string-modifiers` | pure OPY syntax probes for switch, do-while, hex, membership, modifiers, dicts, comprehensions, lambda, and negative diagnostics | -| `compiler-vertical-slice` | minimal OPY HIR to canonical Workshop WIR validation and deterministic emission slice | -| `primitive-lowering` | #46 oracle-backed primitive lowering probe: assignments and modifications (including `**=`), expressions, indexing, format, initializers, implicit default variables at fixed slots; the snapshot constrains the native compiler through structural equivalence | -| `dictionary-lookup` | #145 literal dictionary lookup probe retained from #46: the compiler folds literal key hits and misses to the oracle's selected value or `Null`; dictionary-indexed assignment targets remain a separate boundary | -| `postfix-assignment` | #59 oracle-backed postfix `++`/`--` assignment probe for global, player, and single-level indexed variables; the snapshot constrains native lowering through canonical WIR equivalence | -| `indexed-assignment-nested` | #145 oracle-backed nested indexed-assignment probe for two- and three-level global targets, including compound modification | -| `indexed-assignment-4d-invalid` | #162 pinned negative probe preserving OverPy's structured rejection of four-dimensional indexed assignments at the source semantic boundary | -| `postfix-prefix-invalid` | #59 pinned negative probe for rejected prefix `++` with a stable source-attributed parse diagnostic; prefix `--x` remains valid consecutive unary-minus syntax | -| `postfix-embedded-invalid` | #59 pinned negative probe for the rejected embedded postfix form with a stable source-attributed parse diagnostic | -| `range-player-variable` | #65 pinned oracle-backed player-variable range binder probe for canonical `For Player Variable` lowering | -| `range-invalid-binder` | #162 pinned negative probe for a non-variable range binder with a stable source-attributed semantic diagnostic | -| `member-angle` | #130 catalog-backed `eventPlayer.getHorizontalFacingAngle()` member-value lowering | -| `hud-visibility` | #131 `SpecVisibility.NEVER` to canonical `VISIBLE_NEVER` enum mapping | -| `control-flow-lowering` | #47 pinned oracle-backed control-flow lowering probe: if/elif/else, while, range-for, do-while expansion, switch fallthrough/default, and direct break | -| `switch-break-unsupported` | #47 probe: a break hidden inside a conditional switch arm follows OverPy's `Else` marker lowering, and a top-level no-op switch is elided | -| `switch-order` | #47 pinned oracle probe for a default arm before later case arms and source-order fallthrough | -| `switch-structured-target` | #47 pinned oracle probe for nested if/while structure in an earlier arm and later case/default targets | -| `switch-multiple-break` | #47 pinned oracle probe for multiple direct breaks lowered through canonical nested switch-exit WIR | -| `do-while-break` | #47 pinned oracle probe for direct, conditional, and nested do-while break lowering | -| `do-while-invalid` | #47 pinned negative probe for the stable do-while placement diagnostic | -| `directives`, `duplicate-rule-diagnostic`, `project-main-file` | directive/include/main-file preprocessing probes | -| `switch-break`, `strings-and-lambda`, `lambda-invalid` | switch break/fallthrough, f-string interpolation, and lambda negative probes | -| `receiver-playervar` | bare variable member expression `A = B.C` with preserved receiver/member provenance | - -## Real-world fixtures - -### Derived from upstream OverPy `examples/` (GPL-3.0-only) - -Fixtures whose OPY sources are byte-identical to the pinned reference tree's -`examples/` content (verified by `diff` against the pinned content -commit on acquisition; the fixture `fixture.json` files additionally record -the example-capture commit `eea67adbcf6926c4004e35e25ab4be072624a44e` used -by the original WrightKit acquisition pipeline — both identities describe -the same bytes). Mapping to the pinned tree (content commit -`889d9749d1def17f146548cbddb94ea1ab015847`): - -| Fixture | Pinned-tree source | `expectedStatus` | -| --- | --- | --- | -| `overpy-cake` | `examples/cake.opy` | success | -| `overpy-pixelart` | `examples/pixelart.opy` | success | -| `overpy-santa` | `examples/santa.opy` | success | -| `overpy-cronch` | `examples/cronch.opy` | success | -| `overpy-broken-weapons` | `examples/broken_weapons.opy` | success | -| `overpy-client-to-server` | `examples/clientToServer.opy` | success | -| `overpy-crosshair` | `examples/crosshair.opy` | success | -| `overpy-inputhud` | `examples/inputhud.opy` | success | -| `overpy-parabola` | `examples/parabola.opy` | success | -| `overpy-meipocalypse` | `examples/meipocalypse/*.opy` (the non-OPY generators `generateWalls.js`, `generateZoneVariables.js`, `elements.md`, `todo.md` are not ported) | failure (reference rejects part of the project; recorded in the snapshot) | -| `overpy-zencopter` | `examples/Zencopter/heli.opy` (`heliturrets.js` is not ported) | failure | - -License note: the upstream `examples/` are part of the GPL-3.0-only OverPy -repository. These fixture OPY sources are retained **as provenance-recorded -evidence** for oracle evaluation: they are not redistributed as library code -or imported into the opy-rs core, and the repository's AGPL-3.0-or-later -license does not relicense them. `oracle.json` snapshots record observed -oracle behavior (accept/reject, diagnostics, normalized Workshop text) for -compatibility evidence. See the clean-room policy in -`docs/compatibility/upstream-references.md`. - -Current real-world reference-success/native-gap cases also keep a -minimized regression snippet in the parent fixture's `regressions` metadata. -Those snippets retain a link to the full-project oracle evidence; they are not -standalone replacement expectations. - -`census/workshop-feature-census` is the OPY consumer-side census fixture. Its -`workshopFeatureIds` are opaque IDs reserved for the `workshop-rs#10` contract; -this repository does not copy Workshop catalog definitions or signatures. - -### Independent third-party projects (BSD-2-Clause) - -| Fixture | Origin | `expectedStatus` | -| --- | --- | --- | -| `ow1-emulator` | [Overwatch-1-Emulator/ow1-emulator](https://github.com/Overwatch-1-Emulator/ow1-emulator) `src/1v1_main.opy`, full include closure, pinned commit `25cd6ce8d4acdd64b66c862a55c7ed66c8e50af1`, BSD-2-Clause | failure (recorded reference diagnostics) | -| `6v6-adjustments` | [6v6-Adjustments/6v6-adjustments](https://github.com/6v6-Adjustments/6v6-adjustments) `src/main.opy`, full include closure, pinned dev-branch commit `624480db6b7494f8bd5f3ab68fbb7e96a7726702`, BSD-2-Clause | failure (recorded reference diagnostics) | - -Both are committed unchanged from their pinned commits with per-file SHA-256 -records in `fixture.json` (`files` map). The multi-file fixtures exercise -large include closures, macros, subroutines, settings blocks, and Workshop -enum/action surfaces; their snapshots keep `expectedStatus: failure` with the -reference diagnostics, exactly like the pinned oracle behaves. - -> Note on `acquisitionMethod` in imported `fixture.json` files: imported -> fixtures preserve `scripts/acquire-corpus.py` as the **historical** Wright -> acquisition method. That one-off M11 helper and its corpus manifest were -> retired after live OPY compatibility ownership moved to opy-rs. The field is -> retained only as provenance and is not an executable instruction. Current -> corpus/oracle acquisition and re-validation are owned by this repository; -> do not reintroduce the old Wright helper unless new evidence demonstrates a -> concrete need for an equivalent owner-side acquisition workflow. - -## Not ported / dropped - -* **No fixture was dropped for provenance reasons**: fixtures in the WrightKit - corpus carried complete, reviewed provenance and are ported. -* Upstream `examples/` not ported (candidates for later expansion once a - demonstrated need exists): `lucioball_all_heroes.opy`, `skirmish_elim.opy`, - `settings.opy.json` (settings schema data, not an OPY source), and the - non-OPY generator files listed above. -* The full-gamemode list in the upstream `examples/README.md` (OverWordle, - Riptire Racing, Conquest, …) points to third-party repositories; they stay - out of the corpus until their licenses are reviewed. +`fixture.json` contains: + +- `id`, `category`, optional `features`, and the fixture `source`; +- optional `attribution` with source kind, origin, license, and redistribution + status when a third-party, redistribution, or reproducibility record is needed; +- imported-source `sourceCommit`, `sourceUrl`, `licenseUrl`, and file hashes + where attribution or reproducibility requires them; +- `tests.source` with the native source status, reference relationship, rule + name comparison flag, and stable diagnostic code when applicable; +- `tests.compiler` with the native compiler status, relationship, comparison + contract, and diagnostic details when applicable. + +There is one manifest per fixture. The manifest does not contain Issue/PR +history, run timing, generic verification records, or generated reports. + +`oracle.json` records the pinned oracle identity, the complete resolved OPY +source graph with per-file SHA-256 hashes, compile status, exit code, +diagnostics, normalized Workshop output, and output hash. It is the reference +test result, not a second expectation table. Run `python3 tools/overpy/run_oracle.py` +to verify the snapshots; use `--update` only when intentionally accepting a +new result from the pinned oracle. + +## Fixture groups + +Synthetic fixtures cover OPY syntax, preprocessing, diagnostics, source +semantics, and focused compiler lowering. Real-world fixtures retain complete +projects from the pinned OverPy examples or independently licensed projects; +their `regressions` entries only identify minimized `.opy` files excluded from +the full-project source digest. The complete project remains the integration +test. + +`census/workshop-feature-census` contains opaque Workshop feature IDs used by +the OPY-side consumer test. Catalog definitions, validation, and emission +remain owned by `workshop-rs`. + +Imported source files retain their applicable license, attribution, immutable +source revision, redistribution status, and file hashes. These concrete +records protect licensing and reproducibility; they do not define a generic +compatibility metadata model. diff --git a/crates/opy-rs/tests/fixtures/corpus/census/workshop-feature-census/fixture.json b/crates/opy-rs/tests/fixtures/corpus/census/workshop-feature-census/fixture.json index 8ad74274..686939da 100644 --- a/crates/opy-rs/tests/fixtures/corpus/census/workshop-feature-census/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/census/workshop-feature-census/fixture.json @@ -2,7 +2,12 @@ "schemaVersion": 1, "id": "census/workshop-feature-census", "category": "workshop-census", - "features": ["events", "actions", "values", "control flow"], + "features": [ + "events", + "actions", + "values", + "control flow" + ], "workshopFeatureIds": [ "event.global", "action.disableInspector", @@ -12,18 +17,19 @@ "structural.if" ], "source": "workshop-feature-census.opy", - "expectedStatus": "success", - "provenance": { - "kind": "synthetic-original", - "origin": "WrightKit OPY census boundary fixture", - "license": "AGPL-3.0-or-later", - "redistributable": true, - "modifications": "none" - }, "censusContract": { - "owner": "workshop-rs", - "issue": "https://github.com/wrightkit/workshop-rs/issues/10", - "status": "pending", - "note": "Feature IDs are opaque consumer references; canonical definitions and validation remain owned by workshop-rs." + "owner": "workshop-rs" + }, + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "normalized-output" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/6v6-adjustments/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/6v6-adjustments/fixture.json index 8e0ad1df..51ccc8fe 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/6v6-adjustments/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/6v6-adjustments/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "failure", "features": [ "preprocessing/includes", "declarations", @@ -129,7 +127,14 @@ "utilities/watermark.opy": "7db2d7bcdea9f4bed11f0bd01837acd4127a07cd80cc84a9b465278772fce889" }, "id": "real-world/6v6-adjustments", - "provenance": { + "schemaVersion": 1, + "source": "main.opy", + "regressions": [ + { + "source": "regressions/get-horizontal-facing-angle.opy" + } + ], + "attribution": { "kind": "imported-example", "license": "BSD-2-Clause", "licenseUrl": "https://github.com/6v6-Adjustments/6v6-adjustments/blob/624480db6b7494f8bd5f3ab68fbb7e96a7726702/LICENSE", @@ -139,17 +144,19 @@ "sourceCommit": "624480db6b7494f8bd5f3ab68fbb7e96a7726702", "sourceUrl": "https://github.com/6v6-Adjustments/6v6-adjustments/blob/624480db6b7494f8bd5f3ab68fbb7e96a7726702/src/main.opy" }, - "runtimeSeconds": 9.7, - "schemaVersion": 1, - "source": "main.opy", - "regressions": [ - { - "id": "real-world/6v6-adjustments/get-horizontal-facing-angle", - "source": "regressions/get-horizontal-facing-angle.opy", - "derivedFrom": "heroes/juno/orbital_ray.opy", - "expectedReferenceStatus": "success", - "kind": "minimized-regression", - "provenance": "oracle:real-world/6v6-adjustments/oracle.json; source paths 20-24 in heroes/juno/orbital_ray.opy" + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "unknown-member" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "unknown-member", + "failureClass": "frontend" } - ] + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-broken-weapons/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-broken-weapons/fixture.json index 529eea37..0d3f97f6 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-broken-weapons/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-broken-weapons/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "success", "features": [ "declarations", "macros", @@ -12,7 +10,14 @@ "broken_weapons.opy": "4b966f2b755545143c7a9fafdbb7b5d5e85604910f14d2532d92eea0854ae427" }, "id": "real-world/overpy-broken-weapons", - "provenance": { + "schemaVersion": 1, + "source": "broken_weapons.opy", + "regressions": [ + { + "source": "regressions/numeric-range-setting.opy" + } + ], + "attribution": { "kind": "imported-example", "license": "GPL-3.0-only", "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", @@ -22,17 +27,17 @@ "sourceCommit": "eea67adbcf6926c4004e35e25ab4be072624a44e", "sourceUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/examples/broken_weapons.opy" }, - "runtimeSeconds": 4.4, - "schemaVersion": 1, - "source": "broken_weapons.opy", - "regressions": [ - { - "id": "real-world/overpy-broken-weapons/numeric-range-setting", - "source": "regressions/numeric-range-setting.opy", - "derivedFrom": "broken_weapons.opy", - "expectedReferenceStatus": "success", - "kind": "minimized-regression", - "provenance": "oracle:real-world/overpy-broken-weapons/oracle.json; source path 53" + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "known-gap", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "known-gap", + "comparison": "semantic-wir", + "semanticEquivalent": false } - ] + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-cake/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-cake/fixture.json index ef66ccb5..f3f2603b 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-cake/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-cake/fixture.json @@ -2,10 +2,16 @@ "schemaVersion": 1, "id": "real-world/overpy-cake", "category": "real-world", - "features": ["declarations", "arrays", "expressions", "for", "while", "macros"], + "features": [ + "declarations", + "arrays", + "expressions", + "for", + "while", + "macros" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { + "attribution": { "kind": "imported-example", "origin": "Zezombye/overpy examples/cake.opy", "sourceUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/examples/cake.opy", @@ -14,5 +20,18 @@ "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", "redistributable": true, "modifications": "none" + }, + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "known-gap", + "comparison": "semantic-wir", + "semanticEquivalent": false + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-client-to-server/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-client-to-server/fixture.json index 005a3660..079e3ac4 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-client-to-server/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-client-to-server/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "success", "features": [ "declarations", "expressions", @@ -13,7 +11,20 @@ "clientToServer.opy": "3ef442054f57c1a9f9905f11da4d5e27726564c5bab44c74ebfab56825b94781" }, "id": "real-world/overpy-client-to-server", - "provenance": { + "schemaVersion": 1, + "source": "clientToServer.opy", + "regressions": [ + { + "source": "regressions/chained-ternary.opy" + }, + { + "source": "regressions/is-dummy.opy" + }, + { + "source": "regressions/get-horizontal-facing-angle.opy" + } + ], + "attribution": { "kind": "imported-example", "license": "GPL-3.0-only", "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", @@ -23,33 +34,17 @@ "sourceCommit": "eea67adbcf6926c4004e35e25ab4be072624a44e", "sourceUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/examples/clientToServer.opy" }, - "runtimeSeconds": 6.4, - "schemaVersion": 1, - "source": "clientToServer.opy", - "regressions": [ - { - "id": "real-world/overpy-client-to-server/chained-ternary", - "source": "regressions/chained-ternary.opy", - "derivedFrom": "clientToServer.opy", - "expectedReferenceStatus": "success", - "kind": "minimized-regression", - "provenance": "oracle:real-world/overpy-client-to-server/oracle.json; source path 55" - }, - { - "id": "real-world/overpy-client-to-server/is-dummy", - "source": "regressions/is-dummy.opy", - "derivedFrom": "clientToServer.opy", - "expectedReferenceStatus": "success", - "kind": "minimized-regression", - "provenance": "oracle:real-world/overpy-client-to-server/oracle.json; source path 42" + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "known-gap", + "ruleNames": false }, - { - "id": "real-world/overpy-client-to-server/get-horizontal-facing-angle", - "source": "regressions/get-horizontal-facing-angle.opy", - "derivedFrom": "clientToServer.opy", - "expectedReferenceStatus": "success", - "kind": "minimized-regression", - "provenance": "oracle:real-world/overpy-client-to-server/oracle.json; source paths 46, 58, 60, 62, 64" + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true } - ] + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-cronch/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-cronch/fixture.json index dde78d4c..001fb19b 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-cronch/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-cronch/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "success", "features": [ "declarations", "macros", @@ -13,7 +11,14 @@ "cronch.opy": "413d3d598ae08f3682acf925b7294316a68aba7c9a959e95d66b461485ba9169" }, "id": "real-world/overpy-cronch", - "provenance": { + "schemaVersion": 1, + "source": "cronch.opy", + "regressions": [ + { + "source": "regressions/postfix-increment.opy" + } + ], + "attribution": { "kind": "imported-example", "license": "GPL-3.0-only", "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", @@ -23,17 +28,17 @@ "sourceCommit": "eea67adbcf6926c4004e35e25ab4be072624a44e", "sourceUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/examples/cronch.opy" }, - "runtimeSeconds": 6.1, - "schemaVersion": 1, - "source": "cronch.opy", - "regressions": [ - { - "id": "real-world/overpy-cronch/postfix-increment", - "source": "regressions/postfix-increment.opy", - "derivedFrom": "cronch.opy", - "expectedReferenceStatus": "success", - "kind": "minimized-regression", - "provenance": "oracle:real-world/overpy-cronch/oracle.json; source path 31-34" + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "known-gap", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "known-gap", + "comparison": "semantic-wir", + "semanticEquivalent": false } - ] + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-crosshair/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-crosshair/fixture.json index 95de8766..b1e7928c 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-crosshair/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-crosshair/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "success", "features": [ "settings", "declarations", @@ -13,7 +11,14 @@ "crosshair.opy": "d2c22b59c11fee75ae4febc96c94126059f4e5c5f64cc2e5a28e939968ed8f8f" }, "id": "real-world/overpy-crosshair", - "provenance": { + "schemaVersion": 1, + "source": "crosshair.opy", + "regressions": [ + { + "source": "regressions/byte-string-modifier.opy" + } + ], + "attribution": { "kind": "imported-example", "license": "GPL-3.0-only", "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", @@ -23,17 +28,17 @@ "sourceCommit": "eea67adbcf6926c4004e35e25ab4be072624a44e", "sourceUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/examples/crosshair.opy" }, - "runtimeSeconds": 3.2, - "schemaVersion": 1, - "source": "crosshair.opy", - "regressions": [ - { - "id": "real-world/overpy-crosshair/byte-string-modifier", - "source": "regressions/byte-string-modifier.opy", - "derivedFrom": "crosshair.opy", - "expectedReferenceStatus": "success", - "kind": "minimized-regression", - "provenance": "oracle:real-world/overpy-crosshair/oracle.json; source path 31" + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true } - ] + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-inputhud/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-inputhud/fixture.json index 9092df92..5848bb65 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-inputhud/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-inputhud/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "success", "features": [ "settings", "declarations", @@ -14,7 +12,14 @@ "inputhud.opy": "ae7a0e004cc80e61d256d0ff2509f51bdc96b27d60a93c9215b327436e7eefc1" }, "id": "real-world/overpy-inputhud", - "provenance": { + "schemaVersion": 1, + "source": "inputhud.opy", + "regressions": [ + { + "source": "regressions/implicit-string-concatenation.opy" + } + ], + "attribution": { "kind": "imported-example", "license": "GPL-3.0-only", "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", @@ -24,17 +29,17 @@ "sourceCommit": "eea67adbcf6926c4004e35e25ab4be072624a44e", "sourceUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/examples/inputhud.opy" }, - "runtimeSeconds": 3.2, - "schemaVersion": 1, - "source": "inputhud.opy", - "regressions": [ - { - "id": "real-world/overpy-inputhud/implicit-string-concatenation", - "source": "regressions/implicit-string-concatenation.opy", - "derivedFrom": "inputhud.opy", - "expectedReferenceStatus": "success", - "kind": "minimized-regression", - "provenance": "oracle:real-world/overpy-inputhud/oracle.json; source path 40-44" + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "known-gap", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "known-gap", + "comparison": "semantic-wir", + "semanticEquivalent": false } - ] + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-meipocalypse/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-meipocalypse/fixture.json index fe1d1f57..527e229a 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-meipocalypse/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-meipocalypse/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "failure", "features": [ "declarations", "preprocessing/includes", @@ -24,7 +22,9 @@ "zones.opy": "a4f3d898c31cdcdb427ab02d986d786fa312acd1a00e2cc8dcc2c2f6aac0aba7" }, "id": "real-world/overpy-meipocalypse", - "provenance": { + "schemaVersion": 1, + "source": "meipocalypse.opy", + "attribution": { "kind": "imported-example", "license": "GPL-3.0-only", "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", @@ -34,8 +34,19 @@ "sourceCommit": "eea67adbcf6926c4004e35e25ab4be072624a44e", "sourceUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/examples/meipocalypse/meipocalypse.opy" }, - "provenanceNote": "upstream examples/meipocalypse/ also contains non-.opy generator helpers (generateWalls.js, generateZoneVariables.js, settings.opy.json, *.md) that are intentionally not committed as fixture sources under the compatibility fixture policy (non-.opy sources are not fixture inputs); the .opy sources reference the JS generation steps via __script__ (meipocalypse.opy:57-58, barricades.opy:51), so the pinned oracle fails with ENOENT, recorded as reference evidence in oracle.json and adapter/test/adapter.test.js", - "runtimeSeconds": 4.6, - "schemaVersion": 1, - "source": "meipocalypse.opy" + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "lex-error" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "script-not-found", + "failureClass": "frontend" + } + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-parabola/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-parabola/fixture.json index 4e6bc5b0..0254cb6b 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-parabola/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-parabola/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "success", "features": [ "settings", "declarations", @@ -13,7 +11,14 @@ "parabola.opy": "2dce50c374a00e3d3e664f4753f736f788cfb2d0df3a2149b139e6f8cb484940" }, "id": "real-world/overpy-parabola", - "provenance": { + "schemaVersion": 1, + "source": "parabola.opy", + "regressions": [ + { + "source": "regressions/numeric-enum-member.opy" + } + ], + "attribution": { "kind": "imported-example", "license": "GPL-3.0-only", "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", @@ -23,17 +28,17 @@ "sourceCommit": "eea67adbcf6926c4004e35e25ab4be072624a44e", "sourceUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/examples/parabola.opy" }, - "runtimeSeconds": 3.0, - "schemaVersion": 1, - "source": "parabola.opy", - "regressions": [ - { - "id": "real-world/overpy-parabola/numeric-enum-member", - "source": "regressions/numeric-enum-member.opy", - "derivedFrom": "parabola.opy", - "expectedReferenceStatus": "success", - "kind": "minimized-regression", - "provenance": "oracle:real-world/overpy-parabola/oracle.json; source path 35" + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "known-gap", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "known-gap", + "comparison": "semantic-wir", + "semanticEquivalent": false } - ] + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-pixelart/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-pixelart/fixture.json index 4e6b2240..f5847f1d 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-pixelart/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-pixelart/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "success", "features": [ "declarations", "control flow", @@ -13,7 +11,9 @@ "pixelart.opy": "b9b7c9b0dea9ebd4734a88a10eb8639f32eec8119aefee44333a73cee7ac6502" }, "id": "real-world/overpy-pixelart", - "provenance": { + "schemaVersion": 1, + "source": "pixelart.opy", + "attribution": { "kind": "imported-example", "license": "GPL-3.0-only", "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", @@ -23,7 +23,17 @@ "sourceCommit": "eea67adbcf6926c4004e35e25ab4be072624a44e", "sourceUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/examples/pixelart.opy" }, - "runtimeSeconds": 6.2, - "schemaVersion": 1, - "source": "pixelart.opy" + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "known-gap", + "comparison": "semantic-wir", + "semanticEquivalent": false + } + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-santa/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-santa/fixture.json index b49c299d..722ed40d 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-santa/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-santa/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "success", "features": [ "declarations", "player variables", @@ -13,7 +11,14 @@ "santa.opy": "63d9a92ba2c59423857322c4b6d651a2e86863ed040e8af585599a364f11fe0e" }, "id": "real-world/overpy-santa", - "provenance": { + "schemaVersion": 1, + "source": "santa.opy", + "regressions": [ + { + "source": "regressions/postfix-increment.opy" + } + ], + "attribution": { "kind": "imported-example", "license": "GPL-3.0-only", "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", @@ -23,17 +28,17 @@ "sourceCommit": "eea67adbcf6926c4004e35e25ab4be072624a44e", "sourceUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/examples/santa.opy" }, - "runtimeSeconds": 4.1, - "schemaVersion": 1, - "source": "santa.opy", - "regressions": [ - { - "id": "real-world/overpy-santa/postfix-increment", - "source": "regressions/postfix-increment.opy", - "derivedFrom": "santa.opy", - "expectedReferenceStatus": "success", - "kind": "minimized-regression", - "provenance": "oracle:real-world/overpy-santa/oracle.json; source path 304" + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "known-gap", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "known-gap", + "comparison": "semantic-wir", + "semanticEquivalent": false } - ] + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-zencopter/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-zencopter/fixture.json index 74e09566..649e64ba 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-zencopter/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/overpy-zencopter/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "failure", "features": [ "preprocessing/macros", "expressions", @@ -12,7 +10,9 @@ "heli.opy": "246f74a946b097c299853dc3d46d1a722cbdc6f97baf7e095986cda6963cfe96" }, "id": "real-world/overpy-zencopter", - "provenance": { + "schemaVersion": 1, + "source": "heli.opy", + "attribution": { "kind": "imported-example", "license": "GPL-3.0-only", "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", @@ -22,7 +22,19 @@ "sourceCommit": "eea67adbcf6926c4004e35e25ab4be072624a44e", "sourceUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/examples/Zencopter/heli.opy" }, - "runtimeSeconds": 7.4, - "schemaVersion": 1, - "source": "heli.opy" + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "lex-error" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "lex-error", + "failureClass": "frontend" + } + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/real-world/ow1-emulator/fixture.json b/crates/opy-rs/tests/fixtures/corpus/real-world/ow1-emulator/fixture.json index fc3fc628..eb06c4ed 100644 --- a/crates/opy-rs/tests/fixtures/corpus/real-world/ow1-emulator/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/real-world/ow1-emulator/fixture.json @@ -1,7 +1,5 @@ { - "acquisitionMethod": "scripts/acquire-corpus.py: GitHub archive of the pinned commit (codeload.tar.gz), include-closure resolution matching the pinned oracle, SHA-256 verification", "category": "real-world", - "expectedStatus": "failure", "features": [ "preprocessing/includes", "declarations", @@ -156,7 +154,9 @@ "utilities/watermark.opy": "fcb773484da735c66916280a2d9be7fe3a8dd8db47859ba217bf6135c49ad298" }, "id": "real-world/ow1-emulator", - "provenance": { + "schemaVersion": 1, + "source": "1v1_main.opy", + "attribution": { "kind": "imported-example", "license": "BSD-2-Clause", "licenseUrl": "https://github.com/Overwatch-1-Emulator/ow1-emulator/blob/25cd6ce8d4acdd64b66c862a55c7ed66c8e50af1/LICENSE", @@ -166,7 +166,18 @@ "sourceCommit": "25cd6ce8d4acdd64b66c862a55c7ed66c8e50af1", "sourceUrl": "https://github.com/Overwatch-1-Emulator/ow1-emulator/blob/25cd6ce8d4acdd64b66c862a55c7ed66c8e50af1/src/1v1_main.opy" }, - "runtimeSeconds": 6.8, - "schemaVersion": 1, - "source": "1v1_main.opy" + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "known-gap", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "known-gap", + "comparison": "diagnostic-code", + "diagnosticCode": "unsupported-integration-surface", + "failureClass": "integration" + } + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/array-unique/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/array-unique/fixture.json index f9c55ace..8b93c8c1 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/array-unique/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/array-unique/fixture.json @@ -7,12 +7,23 @@ "compilation/array-unique" ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { + "attribution": { "kind": "original", - "origin": "Minimized from OWBastion/Bastion commit 07b03b1ec587b5cde753939808344359c689082e, src/events/allocation/buildCandidatePool.opy:76, for opy-rs Issue #322", + "origin": "Minimized from OWBastion/Bastion commit 07b03b1ec587b5cde753939808344359c689082e, src/events/allocation/buildCandidatePool.opy:76, for OPY compatibility regression", "license": "AGPL-3.0-or-later", "redistributable": true }, - "provenanceNote": "The literal-array rule pins duplicate removal and first-occurrence order; the player-variable rule preserves the exact Bastion eventTempIndex.unique() receiver shape." + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/basic-rule/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/basic-rule/fixture.json index d6b39e77..75bece38 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/basic-rule/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/basic-rule/fixture.json @@ -3,11 +3,16 @@ "id": "synthetic/basic-rule", "category": "synthetic", "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "normalized-output" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-condition-agentlab/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-condition-agentlab/fixture.json index babb07c4..09dc3ae3 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-condition-agentlab/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-condition-agentlab/fixture.json @@ -8,12 +8,17 @@ "chase over time" ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus", - "license": "AGPL-3.0-or-later", - "redistributable": true - }, - "provenanceNote": "Synthetic-original source authored by Wright for the corpus, reproducing the rule shape of the ChaseTimeReeval.NONE discovery fixture wrightkit/agent-lab tools/test/fixtures/analyzer/chase-condition.opy (blob SHA 2938baff2bac77f8818309ac1e356cf5b9e15d56); agent-lab declares no redistribution license, so its source stays out of the corpus. The pinned oracle compiles this shape (expectedStatus success, with its w_ow2_rule_condition_chase warning recorded in oracle.json). Since #109, Wright's native source implementation resolves chaseOverTime (action) and isGameInProgress (value) through the OPY semantic compatibility manifest (crates/opy-rs/src/manifest) and compiles this fixture end-to-end via the Workshop emission catalog; the manifest entries and their emission spellings are validated by the synthetic manifest probes (crates/opy-rs/src/manifest/probes, #109)." + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-enums/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-enums/fixture.json index c94213de..83833042 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-enums/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-enums/fixture.json @@ -2,14 +2,21 @@ "schemaVersion": 1, "id": "synthetic/chase-enums", "category": "synthetic", - "features": ["workshop enums", "assignments"], + "features": [ + "workshop enums", + "assignments" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus", - "license": "AGPL-3.0-or-later", - "redistributable": true - }, - "provenanceNote": "Synthetic-original source authored by Wright for the corpus. It reproduces the ChaseTimeReeval.NONE usage discovered in wrightkit/agent-lab tools/test/fixtures/analyzer/chase-condition.opy (blob SHA 2938baff2bac77f8818309ac1e356cf5b9e15d56) in a fully supported native-source implementation context (assignments), because agent-lab declares no redistribution license and its source stays out of the corpus. Member spellings and emitted Workshop text were reference-validated against the pinned OverPy 9.7.10 oracle (see docs/language-support.md and crates/wright-workshop/src/catalog/data/catalog.json provenance)." + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "normalized-output" + } + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-keywords/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-keywords/fixture.json index 67c02863..2e87b56e 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-keywords/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/chase-keywords/fixture.json @@ -2,14 +2,23 @@ "schemaVersion": 1, "id": "synthetic/chase-keywords", "category": "synthetic", - "features": ["named/keyword arguments", "chase forms", "ChaseReeval contextual resolution"], + "features": [ + "named/keyword arguments", + "chase forms", + "ChaseReeval contextual resolution" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus", - "license": "AGPL-3.0-or-later", - "redistributable": true - }, - "provenanceNote": "Synthetic-original source authored by Wright for the corpus. It reproduces the chase(...) keyword forms discovered in the real-world overpy-meipocalypse corpus (chase(variable, destination, rate=..., ChaseReeval.NONE), e.g. mei_types.opy and fightforyourlife.opy) in a fully supported native-source implementation context, and exercises the generic keyword-argument binding for wait/vect/len/print/getPlayersInRadius/setStatusEffect. Reference-validated against the pinned OverPy 9.7.10 oracle (issue #110); the meipocalypse fixture itself stays out of the corpus because its broader surface (list comprehensions, Team.1, script hooks) is not native-supported." + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-invalid/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-invalid/fixture.json index 94852f32..69189b95 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-invalid/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-invalid/fixture.json @@ -2,13 +2,25 @@ "schemaVersion": 1, "id": "synthetic/collection-mutation-328-invalid", "category": "compilation", - "features": ["del", "nested-arrays", "semantic-boundary"], + "features": [ + "del", + "nested-arrays", + "semantic-boundary" + ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #328 pinned OverPy 9.7.10 five-index deletion rejection probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "four-dimensional-delete" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "four-dimensional-delete", + "failureClass": "frontend" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-random-invalid/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-random-invalid/fixture.json index bb38a4a4..a7ef59ba 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-random-invalid/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-random-invalid/fixture.json @@ -2,13 +2,25 @@ "schemaVersion": 1, "id": "synthetic/collection-mutation-328-random-invalid", "category": "compilation", - "features": ["del", "nested-arrays", "random-index", "semantic-boundary"], + "features": [ + "del", + "nested-arrays", + "random-index", + "semantic-boundary" + ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #328 pinned OverPy 9.7.10 nested deletion random-index rejection probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "known-gap", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "unsupported-integration-surface", + "failureClass": "integration" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-random-player-invalid/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-random-player-invalid/fixture.json index f225ea02..bd06fa06 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-random-player-invalid/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328-random-player-invalid/fixture.json @@ -2,13 +2,25 @@ "schemaVersion": 1, "id": "synthetic/collection-mutation-328-random-player-invalid", "category": "compilation", - "features": ["del", "nested-arrays", "random-player-receiver", "semantic-boundary"], + "features": [ + "del", + "nested-arrays", + "random-player-receiver", + "semantic-boundary" + ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #328 pinned OverPy 9.7.10 nested deletion random-player receiver rejection probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "known-gap", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "unsupported-integration-surface", + "failureClass": "integration" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328/fixture.json index 2858d672..98e14b22 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/collection-mutation-328/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/collection-mutation-328", "category": "compilation", - "features": ["del", "indexed-mutation", "nested-arrays", "player-variables"], + "features": [ + "del", + "indexed-mutation", + "nested-arrays", + "player-variables" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #328 minimized pinned OverPy 9.7.10 collection-deletion probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/compiler-structure/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/compiler-structure/fixture.json index c5caecad..1e3db2c5 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/compiler-structure/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/compiler-structure/fixture.json @@ -2,13 +2,21 @@ "schemaVersion": 1, "id": "synthetic/compiler-structure", "category": "synthetic", - "features": ["compilation/opy-structural-lowering"], + "features": [ + "compilation/opy-structural-lowering" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #40 minimized structural lowering probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/compiler-vertical-slice/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/compiler-vertical-slice/fixture.json index 90c651e3..a7b7afd1 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/compiler-vertical-slice/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/compiler-vertical-slice/fixture.json @@ -3,14 +3,19 @@ "id": "synthetic/compiler-vertical-slice", "category": "synthetic", "source": "source.opy", - "expectedStatus": "success", "features": [ "compilation/opy-integration-vertical-slice" ], - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus; OPY-to-Workshop integration boundary issue #35", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "normalized-output" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/conditional-forward-gotos/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/conditional-forward-gotos/fixture.json index 0fa9b527..a14caa0d 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/conditional-forward-gotos/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/conditional-forward-gotos/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/conditional-forward-gotos", "category": "compilation", - "features": ["if", "else", "goto", "forward-control-flow"], + "features": [ + "if", + "else", + "goto", + "forward-control-flow" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #319 review regression probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow-328/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow-328/fixture.json index 2864f42c..5f6e4032 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow-328/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow-328/fixture.json @@ -2,13 +2,26 @@ "schemaVersion": 1, "id": "synthetic/control-flow-328", "category": "compilation", - "features": ["continue", "do-while", "goto", "loc-plus", "rule-start", "switch-in-loop"], + "features": [ + "continue", + "do-while", + "goto", + "loc-plus", + "rule-start", + "switch-in-loop" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #328 minimized pinned OverPy 9.7.10 control-flow probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow-lowering/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow-lowering/fixture.json index da7e0c8a..9d6b010a 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow-lowering/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow-lowering/fixture.json @@ -2,13 +2,27 @@ "schemaVersion": 1, "id": "synthetic/control-flow-lowering", "category": "compilation", - "features": ["control flow", "if", "while", "for", "doWhile", "switch", "break"], + "features": [ + "control flow", + "if", + "while", + "for", + "doWhile", + "switch", + "break" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #47 pinned control-flow lowering probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow/fixture.json index e0c7f236..bb7480d6 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/control-flow/fixture.json @@ -2,13 +2,25 @@ "schemaVersion": 1, "id": "synthetic/control-flow", "category": "synthetic", - "features": ["for", "while", "if", "elif", "else"], + "features": [ + "for", + "while", + "if", + "elif", + "else" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/declarations-numbers/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/declarations-numbers/fixture.json index c9335dea..8707c414 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/declarations-numbers/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/declarations-numbers/fixture.json @@ -3,11 +3,16 @@ "id": "synthetic/declarations-numbers", "category": "synthetic", "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "normalized-output" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/declarations-rules/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/declarations-rules/fixture.json index 97422f79..2e4aed51 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/declarations-rules/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/declarations-rules/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/declarations-rules", "category": "synthetic", - "features": ["global variables", "player variables", "subroutines", "rules"], + "features": [ + "global variables", + "player variables", + "subroutines", + "rules" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/diagnostics/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/diagnostics/fixture.json index 2bd84a8e..73bb4117 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/diagnostics/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/diagnostics/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/diagnostics", "category": "synthetic", - "features": ["syntax diagnostic", "failure result"], + "features": [ + "syntax diagnostic", + "failure result" + ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "parse-error" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "parse-error", + "failureClass": "frontend" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/dictionary-lookup/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/dictionary-lookup/fixture.json index 824e9d18..f6cb9090 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/dictionary-lookup/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/dictionary-lookup/fixture.json @@ -6,11 +6,17 @@ "compilation/opy-literal-dict-lookup" ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #145 literal dictionary lookup lowering probe, retained from the Issue #46 source fixture", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/directives-scoped/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/directives-scoped/fixture.json index 44771e59..ef082929 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/directives-scoped/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/directives-scoped/fixture.json @@ -2,13 +2,23 @@ "schemaVersion": 1, "id": "synthetic/directives-scoped", "category": "preprocessing", - "features": ["preprocessing/rule-prefix-template", "preprocessing/macro-redeclaration", "preprocessing/translations"], + "features": [ + "preprocessing/rule-prefix-template", + "preprocessing/macro-redeclaration", + "preprocessing/translations" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #31 minimized pinned-oracle positive probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/directives/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/directives/fixture.json index 5c694717..60b9437b 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/directives/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/directives/fixture.json @@ -2,52 +2,30 @@ "schemaVersion": 1, "id": "synthetic/directives", "category": "preprocessing", - "features": ["preprocessing/advanced-directives", "preprocessing/translations", "preprocessing/optimization", "annotations/rule-state"], + "features": [ + "preprocessing/advanced-directives", + "preprocessing/translations", + "preprocessing/optimization", + "annotations/rule-state" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #29 minimized pinned-oracle directive and annotation probe", - "license": "AGPL-3.0-or-later", - "redistributable": true - }, "regressions": [ { - "id": "synthetic/directives/disabled-bastion", - "source": "regressions/disabled-bastion.opy", - "derivedFrom": "OWBastion/Bastion#214 disabled-rule differential", - "expectedReferenceStatus": "success", - "kind": "minimized-regression", - "provenance": { - "evidenceUrl": "https://github.com/OWBastion/Bastion/pull/214#issuecomment-5690854871", - "sourceRepository": "https://github.com/OWBastion/Bastion", - "sourceRevision": "f95d3159effed8aef0747a9fac5495e0651895b6", - "entrypoints": ["src/main.opy", "src/externalMain.opy"], - "sourcePaths": [ - "src/composition/bootstrap.opy", - "src/effects/player.opy", - "src/events/effects/buff/gale_blessing.opy", - "src/events/effects/buff/phase_surge.opy", - "src/heroes/juno.opy", - "src/heroes/symmetra.opy", - "src/heroes/zenyatta.opy", - "src/utilities/dev_support/devTool.opy" - ], - "overlay": { - "appliedTo": "every .opy file in the entrypoint include closure", - "replacements": [ - { - "directive": "#!optimizeStrict", - "replacement": "comment" - }, - { - "directive": "#!useVariableForCompressionAlphabet", - "replacement": "comment" - } - ], - "reason": "Known opy-rs backend-directive-unsupported gap (#271); unrelated to disabled-rule lowering" - } - } + "source": "regressions/disabled-bastion.opy" } - ] + ], + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "unsupported", + "comparison": "diagnostic-code", + "diagnosticCode": "backend-directive-unsupported", + "failureClass": "integration" + } + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/do-while-break/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/do-while-break/fixture.json index 0f4a147e..7700c740 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/do-while-break/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/do-while-break/fixture.json @@ -2,13 +2,23 @@ "schemaVersion": 1, "id": "synthetic/do-while-break", "category": "compilation", - "features": ["doWhile", "break", "nested-doWhile"], + "features": [ + "doWhile", + "break", + "nested-doWhile" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #47 pinned do-while break-shape lowering probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/do-while-invalid/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/do-while-invalid/fixture.json index 24d9d25b..c1ee1b69 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/do-while-invalid/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/do-while-invalid/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/do-while-invalid", "category": "diagnostics", - "features": ["doWhile", "placement-diagnostic"], + "features": [ + "doWhile", + "placement-diagnostic" + ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #47 pinned do-while placement diagnostic probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "do-while-placement" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "do-while-placement", + "failureClass": "frontend" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/duplicate-rule-diagnostic/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/duplicate-rule-diagnostic/fixture.json index 0aa4301d..f9317164 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/duplicate-rule-diagnostic/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/duplicate-rule-diagnostic/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/duplicate-rule-diagnostic", "category": "preprocessing", - "features": ["preprocessing/directive-diagnostics", "annotations/directive-diagnostics"], + "features": [ + "preprocessing/directive-diagnostics", + "annotations/directive-diagnostics" + ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #29 malformed directive and annotation probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "duplicate-rule-name" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "duplicate-rule-name", + "failureClass": "frontend" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/expressions-values/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/expressions-values/fixture.json index 161dcdde..81f1026b 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/expressions-values/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/expressions-values/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/expressions-values", "category": "synthetic", - "features": ["expressions", "arrays", "strings", "vectors"], + "features": [ + "expressions", + "arrays", + "strings", + "vectors" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/for-range-agentlab/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/for-range-agentlab/fixture.json index b745152d..5bbf644c 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/for-range-agentlab/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/for-range-agentlab/fixture.json @@ -2,13 +2,28 @@ "schemaVersion": 1, "id": "synthetic/for-range-agentlab", "category": "synthetic", - "features": ["for", "range", "implicit-default-var-binder"], + "features": [ + "for", + "range", + "implicit-default-var-binder" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { + "attribution": { "kind": "derived", "origin": "wrightkit/agent-lab tools/test/fixtures/analyzer/loop-for.opy (default-branch commit at 2026-08-15); the @Condition getTotalTimeElapsed() > 5 line is dropped because that builtin is outside the declared manifest surface, keeping the for-binder regression itself intact", "license": "AGPL-3.0-or-later", "redistributable": true + }, + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "normalized-output" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/hud-subheader/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/hud-subheader/fixture.json index 3957e418..23ae9d19 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/hud-subheader/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/hud-subheader/fixture.json @@ -1,18 +1,23 @@ { "category": "synthetic", - "expectedStatus": "success", "features": [ "hudSubheader special lowering", "canonical HUD reevaluation and spectator visibility" ], "id": "synthetic/hud-subheader", - "provenance": { - "kind": "minimized-regression", - "license": "AGPL-3.0-or-later", - "modifications": "WrightKit-authored minimized reproduction from wrightkit/opy-rs#114", - "origin": "wrightkit/opy-rs#114 acceptance reproduction", - "redistributable": true - }, "schemaVersion": 1, - "source": "source.opy" + "source": "source.opy", + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/hud-visibility/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/hud-visibility/fixture.json index df948449..fb1ead38 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/hud-visibility/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/hud-visibility/fixture.json @@ -7,11 +7,17 @@ "hudSubheader spectator visibility" ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "minimized-regression", - "origin": "wrightkit/opy-rs#131 acceptance reproduction", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/include-scope/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/include-scope/fixture.json index 5a447606..88b2a5b8 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/include-scope/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/include-scope/fixture.json @@ -2,13 +2,22 @@ "schemaVersion": 1, "id": "synthetic/include-scope", "category": "preprocessing", - "features": ["preprocessing/scoped-state", "preprocessing/include-restoration", "preprocessing/optimization"], + "features": [ + "preprocessing/scoped-state", + "preprocessing/include-restoration", + "preprocessing/optimization" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #31 minimized pinned-oracle nested-scope probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "compiler-contract" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/indexed-assignment-4d-invalid/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/indexed-assignment-4d-invalid/fixture.json index f857ee07..5ee26f99 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/indexed-assignment-4d-invalid/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/indexed-assignment-4d-invalid/fixture.json @@ -2,13 +2,23 @@ "schemaVersion": 1, "id": "synthetic/indexed-assignment-4d-invalid", "category": "compilation", - "features": ["compilation/opy-assignment-lowering"], + "features": [ + "compilation/opy-assignment-lowering" + ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #60 pinned four-dimensional indexed-assignment rejection probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "four-dimensional-assignment" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "four-dimensional-assignment", + "failureClass": "frontend" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/indexed-assignment-nested/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/indexed-assignment-nested/fixture.json index 406e508b..b7d9ae82 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/indexed-assignment-nested/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/indexed-assignment-nested/fixture.json @@ -2,13 +2,21 @@ "schemaVersion": 1, "id": "synthetic/indexed-assignment-nested", "category": "compilation", - "features": ["compilation/opy-assignment-lowering"], + "features": [ + "compilation/opy-assignment-lowering" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #60 nested indexed-assignment probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/lambda-invalid/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/lambda-invalid/fixture.json index fdf09458..404b637a 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/lambda-invalid/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/lambda-invalid/fixture.json @@ -2,13 +2,23 @@ "schemaVersion": 1, "id": "synthetic/lambda-invalid", "category": "diagnostics", - "features": ["syntax/lambda"], + "features": [ + "syntax/lambda" + ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #33 pinned negative lambda-context oracle probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "lambda-context" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "lambda-context", + "failureClass": "frontend" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/legacy-entry-include/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/legacy-entry-include/fixture.json index 07ff154f..633a65d2 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/legacy-entry-include/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/legacy-entry-include/fixture.json @@ -2,13 +2,21 @@ "schemaVersion": 1, "id": "synthetic/legacy-entry-include", "category": "preprocessing", - "features": ["preprocessing/include-scope", "preprocessing/file-provenance"], + "features": [ + "preprocessing/include-scope", + "preprocessing/file-provenance" + ], "source": "main.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "WrightKit-authored minimized regression from the OWBastion/Bastion entry include topology", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "compiler-contract" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/member-angle/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/member-angle/fixture.json index dfd890d8..bc1eaf32 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/member-angle/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/member-angle/fixture.json @@ -7,11 +7,17 @@ "compilation/catalog-backed-member-values" ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "minimized-regression", - "origin": "wrightkit/opy-rs#130 acceptance reproduction", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/member-values/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/member-values/fixture.json index 1764ed54..10c2c1ea 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/member-values/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/member-values/fixture.json @@ -7,11 +7,17 @@ "compilation/catalog-backed-member-values" ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #113 minimized eventPlayer.isDummy() lowering probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/optimize-strict/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/optimize-strict/fixture.json index 2c5518bd..043c0a58 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/optimize-strict/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/optimize-strict/fixture.json @@ -2,13 +2,22 @@ "schemaVersion": 1, "id": "synthetic/optimize-strict", "category": "compiler", - "features": ["compiler/optimization", "preprocessing/optimization"], + "features": [ + "compiler/optimization", + "preprocessing/optimization" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #288 minimized pinned-oracle strict optimizer probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-assignment/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-assignment/fixture.json index 448b343c..6ea44a3f 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-assignment/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-assignment/fixture.json @@ -7,11 +7,17 @@ "compilation/opy-assignment-lowering" ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #59 bounded postfix increment and decrement assignment probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-embedded-invalid/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-embedded-invalid/fixture.json index 1c7aa60e..710df8e6 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-embedded-invalid/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-embedded-invalid/fixture.json @@ -6,11 +6,19 @@ "syntax/postfix-assignment" ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #59 embedded postfix assignment rejection probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "parse-error" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "parse-error", + "failureClass": "frontend" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-prefix-invalid/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-prefix-invalid/fixture.json index e9b5d1f0..d66e5cb5 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-prefix-invalid/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/postfix-prefix-invalid/fixture.json @@ -6,11 +6,19 @@ "syntax/postfix-assignment" ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #59 prefix postfix assignment rejection probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "parse-error" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "parse-error", + "failureClass": "frontend" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/preprocessing/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/preprocessing/fixture.json index f8ff06e7..919ea402 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/preprocessing/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/preprocessing/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/preprocessing", "category": "synthetic", - "features": ["include", "macro", "enum", "compiler directives"], + "features": [ + "include", + "macro", + "enum", + "compiler directives" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "known-gap", + "comparison": "semantic-wir", + "semanticEquivalent": false + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/primitive-lowering/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/primitive-lowering/fixture.json index 300cb74c..9be24fff 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/primitive-lowering/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/primitive-lowering/fixture.json @@ -7,11 +7,17 @@ "compilation/opy-expression-lowering" ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #46 non-control-flow statement and value primitive lowering probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/project-entry/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/project-entry/fixture.json index 39a68932..e07270b2 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/project-entry/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/project-entry/fixture.json @@ -2,13 +2,21 @@ "schemaVersion": 1, "id": "synthetic/project-entry", "category": "preprocessing", - "features": ["preprocessing/main-file", "preprocessing/include-scope"], + "features": [ + "preprocessing/main-file", + "preprocessing/include-scope" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #129 included-file mainFile scope probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "normalized-output" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/project-main-file/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/project-main-file/fixture.json index 80586d7a..be976d75 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/project-main-file/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/project-main-file/fixture.json @@ -2,13 +2,21 @@ "schemaVersion": 1, "id": "synthetic/project-main-file", "category": "preprocessing", - "features": ["preprocessing/main-file", "preprocessing/include-scope"], + "features": [ + "preprocessing/main-file", + "preprocessing/include-scope" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #29 mainFile and child-include scope probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "normalized-output" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/range-invalid-binder/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/range-invalid-binder/fixture.json index 3c7893aa..d0b242c7 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/range-invalid-binder/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/range-invalid-binder/fixture.json @@ -2,13 +2,25 @@ "schemaVersion": 1, "id": "synthetic/range-invalid-binder", "category": "diagnostics", - "features": ["for", "range", "invalid-binder-diagnostic"], + "features": [ + "for", + "range", + "invalid-binder-diagnostic" + ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #65 pinned OverPy invalid range-for binder probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "invalid-range-binder" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "invalid-range-binder", + "failureClass": "frontend" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/range-player-variable/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/range-player-variable/fixture.json index b0c09323..f03a68cf 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/range-player-variable/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/range-player-variable/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/range-player-variable", "category": "compilation", - "features": ["for", "range", "player-variable-binder", "host-player-receiver"], + "features": [ + "for", + "range", + "player-variable-binder", + "host-player-receiver" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #65 pinned OverPy player-variable range-for probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/receiver-calls/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/receiver-calls/fixture.json index e520429e..dbaadd82 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/receiver-calls/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/receiver-calls/fixture.json @@ -10,15 +10,25 @@ "player variable receiver" ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { + "files": { + "source.opy": "72ab2b077545fea887e26bd0544441e42bbaab569e5fdd77a9650452e08982e9" + }, + "attribution": { "kind": "original", "origin": "Wright compatibility corpus; receiver/member call forms derived from the real-world overpy-meipocalypse corpus (Zezombye/overpy examples/meipocalypse at commit eea67adbcf6926c4004e35e25ab4be072624a44e, GPL-3.0-only, see real-world/overpy-meipocalypse fixture.json)", "license": "AGPL-3.0-or-later", "redistributable": true }, - "provenanceNote": "Synthetic-original fixture: newly authored for the issue #104 regression. The receiver-call idioms (eventPlayer.setMoveSpeed(...), eventPlayer.getPosition(), eventPlayer.isAlive(), eventPlayer.getHealth(), eventPlayer.teleport(...), and the player-variable receiver) are the same forms exercised by the redistributable overpy-meipocalypse corpus; unsupported enum-gated methods from that corpus (setInvisibility/setStatusEffect with Invis/Status enums, worldVector with Transform) are intentionally excluded and remain structured diagnostics. Oracle en-US spellings for the exercised methods come from the pinned overpy@9.7.10 package surface.", - "files": { - "source.opy": "72ab2b077545fea887e26bd0544441e42bbaab569e5fdd77a9650452e08982e9" + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "normalized-output" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/receiver-playervar/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/receiver-playervar/fixture.json index d99facea..d21383a1 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/receiver-playervar/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/receiver-playervar/fixture.json @@ -1,6 +1,5 @@ { "category": "synthetic", - "expectedStatus": "success", "features": [ "semantics/receiver-playervar" ], @@ -8,13 +7,19 @@ "source.opy": "c8040ee474759c5b27541b2b9a514d350f605f471bc23246d1baeadc37a5eadc" }, "id": "synthetic/receiver-playervar", - "provenance": { - "kind": "original", - "license": "AGPL-3.0-or-later", - "origin": "Wright compatibility corpus; minimized source-language regression for the pinned OverPy receiver-member behavior.", - "redistributable": true - }, - "provenanceNote": "Synthetic-original fixture authored from the pinned oracle observation that a bare variable member expression is accepted. It does not reproduce Workshop catalog data; member existence remains an integration concern.", "schemaVersion": 1, - "source": "source.opy" + "source": "source.opy", + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } + } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/settings/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/settings/fixture.json index 6ee9dd18..251e5c44 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/settings/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/settings/fixture.json @@ -2,13 +2,23 @@ "schemaVersion": 1, "id": "synthetic/settings", "category": "synthetic", - "features": ["settings", "declarations", "rules"], + "features": [ + "settings", + "declarations", + "rules" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "Wright compatibility corpus", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/string-modifiers/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/string-modifiers/fixture.json index 8e9230c9..a7cab2b1 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/string-modifiers/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/string-modifiers/fixture.json @@ -2,13 +2,21 @@ "schemaVersion": 1, "id": "synthetic/string-modifiers", "category": "syntax", - "features": ["syntax/string-modifiers"], + "features": [ + "syntax/string-modifiers" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #28 minimized pinned-oracle string modifier probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/strings-and-lambda/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/strings-and-lambda/fixture.json index 4fd2817c..bfd5e787 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/strings-and-lambda/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/strings-and-lambda/fixture.json @@ -2,13 +2,22 @@ "schemaVersion": 1, "id": "synthetic/strings-and-lambda", "category": "syntax", - "features": ["syntax/string-modifiers", "syntax/lambda"], + "features": [ + "syntax/string-modifiers", + "syntax/lambda" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #33 pinned f-string interpolation and lambda-slot probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-break-unsupported/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-break-unsupported/fixture.json index 9183d7ca..bfab7164 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-break-unsupported/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-break-unsupported/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/switch-break-unsupported", "category": "compilation", - "features": ["control flow", "switch", "break", "negative diagnostic"], + "features": [ + "control flow", + "switch", + "break", + "negative diagnostic" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #47 nested conditional switch-break negative probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-break/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-break/fixture.json index c8ae8277..0cef729f 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-break/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-break/fixture.json @@ -2,13 +2,22 @@ "schemaVersion": 1, "id": "synthetic/switch-break", "category": "syntax", - "features": ["syntax/switch", "syntax/break"], + "features": [ + "syntax/switch", + "syntax/break" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #33 pinned switch fallthrough and break probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-multiple-break/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-multiple-break/fixture.json index b9bf96c5..beed3094 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-multiple-break/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-multiple-break/fixture.json @@ -2,14 +2,25 @@ "schemaVersion": 1, "id": "synthetic/switch-multiple-break", "category": "compilation", - "features": ["switch", "case", "default", "break"], + "features": [ + "switch", + "case", + "default", + "break" + ], "source": "source.opy", "semanticOracle": "semantic-oracle.json", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #47 pinned multi-break lowering probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-order/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-order/fixture.json index 301d099e..02d32e29 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-order/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-order/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/switch-order", "category": "compilation", - "features": ["switch", "case", "default", "fallthrough"], + "features": [ + "switch", + "case", + "default", + "fallthrough" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #47 pinned switch-order lowering probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-structured-target/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-structured-target/fixture.json index 2a271f96..611719cc 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-structured-target/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/switch-structured-target/fixture.json @@ -2,13 +2,24 @@ "schemaVersion": 1, "id": "synthetic/switch-structured-target", "category": "compilation", - "features": ["switch", "case", "default", "nested-structured-control-flow"], + "features": [ + "switch", + "case", + "default", + "nested-structured-control-flow" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #47 pinned structured switch-target lowering probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/syntax-invalid/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/syntax-invalid/fixture.json index 5306fc75..9b13329c 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/syntax-invalid/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/syntax-invalid/fixture.json @@ -2,13 +2,25 @@ "schemaVersion": 1, "id": "synthetic/syntax-invalid", "category": "diagnostics", - "features": ["syntax/do-while", "syntax/string-modifiers", "syntax/dicts"], + "features": [ + "syntax/do-while", + "syntax/string-modifiers", + "syntax/dicts" + ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #28 minimized reference-rejected syntax probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "parse-error" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "parse-error", + "failureClass": "frontend" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/syntax-surface/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/syntax-surface/fixture.json index b20936b7..77879da4 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/syntax-surface/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/syntax-surface/fixture.json @@ -13,11 +13,17 @@ "syntax/lambda" ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #28 minimized pure-OPY syntax probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": true + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/translations-invalid/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/translations-invalid/fixture.json index 37f62f56..f246fc9a 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/translations-invalid/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/translations-invalid/fixture.json @@ -2,13 +2,23 @@ "schemaVersion": 1, "id": "synthetic/translations-invalid", "category": "preprocessing", - "features": ["preprocessing/translations"], + "features": [ + "preprocessing/translations" + ], "source": "source.opy", - "expectedStatus": "failure", - "provenance": { - "kind": "original", - "origin": "opy-rs Issue #31 minimized pinned-oracle negative probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "failure", + "relationship": "match", + "ruleNames": false, + "diagnosticCode": "translations-invalid" + }, + "compiler": { + "nativeStatus": "failure", + "relationship": "match", + "comparison": "diagnostic-code", + "diagnosticCode": "translations-invalid", + "failureClass": "frontend" + } } } diff --git a/crates/opy-rs/tests/fixtures/corpus/synthetic/wait-optimization/fixture.json b/crates/opy-rs/tests/fixtures/corpus/synthetic/wait-optimization/fixture.json index 315460a0..f13fc925 100644 --- a/crates/opy-rs/tests/fixtures/corpus/synthetic/wait-optimization/fixture.json +++ b/crates/opy-rs/tests/fixtures/corpus/synthetic/wait-optimization/fixture.json @@ -2,13 +2,23 @@ "schemaVersion": 1, "id": "synthetic/wait-optimization", "category": "compilation", - "features": ["wait", "optimizeForSize", "default arguments"], + "features": [ + "wait", + "optimizeForSize", + "default arguments" + ], "source": "source.opy", - "expectedStatus": "success", - "provenance": { - "kind": "minimized-regression", - "origin": "opy-rs Issue #282 pinned OverPy 9.7.10 wait lowering probe", - "license": "AGPL-3.0-or-later", - "redistributable": true + "tests": { + "source": { + "nativeStatus": "success", + "relationship": "match", + "ruleNames": false + }, + "compiler": { + "nativeStatus": "success", + "relationship": "match", + "comparison": "semantic-wir", + "semanticEquivalent": true + } } } diff --git a/docs/README.md b/docs/README.md index 547f28c7..a56e04a8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,8 +10,8 @@ architecture/README.md current architecture routing ├─ language-core.md current OverPy semantic/scope contract └─ workshop-boundary.md current canonical Workshop boundary adr/ historical architecture decisions -language-support.md current evidenced support -compatibility/ pinned reference/provenance +language-support.md current tested support +compatibility/ pinned reference/licensing opy/ + hir/ public/API/provider and implementation notes source/tests/corpus current implementation reality Issues / PRs / releases mutable execution state @@ -20,7 +20,7 @@ Issues / PRs / releases mutable execution state For substantive implementation work, start from [`architecture/README.md`](architecture/README.md), then inspect the relevant source/tests and Issue contract. Current support is established by executable -evidence and [`language-support.md`](language-support.md), not by architecture +tests and [`language-support.md`](language-support.md), not by architecture intent alone. ## Current architecture @@ -32,7 +32,7 @@ intent alone. - [OverPy / Workshop boundary](architecture/workshop-boundary.md): lowering, canonical WIR, reconstruction, and dependency direction. - [Repository agent guidance](../AGENTS.md): implementation preflight, - provenance, validation, and delivery. + source attribution, validation, and delivery. - [ADR registry](adr/README.md): historical decisions and their rationale; current contracts remain under `docs/architecture/`. @@ -42,21 +42,21 @@ compatibility pointers rather than separate architecture authorities. ## Compatibility and current support -- [OverPy support contract](language-support.md): current evidenced feature +- [OverPy support contract](language-support.md): current tested feature coverage. - [Upstream references](compatibility/upstream-references.md): pinned upstream - identity, provenance, licensing, and reference boundaries. -- [OverPy evidence harness](../tools/overpy/README.md): probes/snapshots and + identity, source attribution, licensing, and reference boundaries. +- [OverPy compatibility harness](../tools/overpy/README.md): tests/snapshots and differential testing. -Evidence describes implementation completeness; it does not +Tests describe implementation completeness; they do not narrow the established upstream core-language scope. ## APIs and implementation notes - [Tooling API](opy/tooling-api.md): standalone Rust and CLI contracts. - [LPP provider](opy/provider.md): integration/process contract. -- [Source-edit policy](opy/trivia-retention-policy.md): provenance/trivia +- [Source-edit policy](opy/trivia-retention-policy.md): source mapping/trivia requirements. - [Opy HIR v2](hir/opy-hir-v2.md): current HIR representation/wire contract, subject to the current architecture contracts and code reality. diff --git a/docs/adr/0001-public-rust-embedding-surface.md b/docs/adr/0001-public-rust-embedding-surface.md index 253c8b76..f70e1558 100644 --- a/docs/adr/0001-public-rust-embedding-surface.md +++ b/docs/adr/0001-public-rust-embedding-surface.md @@ -2,7 +2,7 @@ - Status: Accepted - Date: 2026-09-12 (backfilled) -- Historical evidence: [Issue #111](https://github.com/wrightkit/opy-rs/issues/111), [PR #112](https://github.com/wrightkit/opy-rs/pull/112) +- Related history: [Issue #111](https://github.com/wrightkit/opy-rs/issues/111), [PR #112](https://github.com/wrightkit/opy-rs/pull/112) ## Context diff --git a/docs/adr/0002-independent-conformance-evidence.md b/docs/adr/0002-independent-conformance-evidence.md index 1ac31757..0ec3fc26 100644 --- a/docs/adr/0002-independent-conformance-evidence.md +++ b/docs/adr/0002-independent-conformance-evidence.md @@ -1,8 +1,11 @@ # ADR-0002: Independent offline OverPy conformance evidence -- Status: Accepted +- Status: Superseded by [ADR-0006](0006-tests-first-compatibility.md) - Date: 2026-09-12 (backfilled) -- Historical evidence: [Issue #158](https://github.com/wrightkit/opy-rs/issues/158), [PR #159](https://github.com/wrightkit/opy-rs/pull/159) +- Related history: [Issue #158](https://github.com/wrightkit/opy-rs/issues/158), [PR #159](https://github.com/wrightkit/opy-rs/pull/159) + +This record preserves the historical decision. The current compatibility +contract is defined by [ADR-0006](0006-tests-first-compatibility.md). ## Context diff --git a/docs/adr/0003-first-party-provider-boundary.md b/docs/adr/0003-first-party-provider-boundary.md index 8fa68ab7..de70a145 100644 --- a/docs/adr/0003-first-party-provider-boundary.md +++ b/docs/adr/0003-first-party-provider-boundary.md @@ -2,7 +2,7 @@ - Status: Accepted - Date: 2026-09-12 (backfilled) -- Historical evidence: [Issue #170](https://github.com/wrightkit/opy-rs/issues/170), [PR #171](https://github.com/wrightkit/opy-rs/pull/171), [LPP entry-loading contract](https://github.com/wrightkit/language-provider-protocol/issues/16) +- Related history: [Issue #170](https://github.com/wrightkit/opy-rs/issues/170), [PR #171](https://github.com/wrightkit/opy-rs/pull/171), [LPP entry-loading contract](https://github.com/wrightkit/language-provider-protocol/issues/16) ## Context @@ -16,7 +16,7 @@ make diagnostics and source identity diverge from the standalone library. The first-party LPP provider is a thin owner process over `opy-rs`. It accepts the selected entry through the approved LPP contract and delegates project loading, `#!mainFile`, reachable includes, preprocessing, macros, source -semantics, diagnostics/provenance, and compilation to `opy-rs`. It advertises +semantics, diagnostics/source mapping, and compilation to `opy-rs`. It advertises only the capabilities its implementation supports and returns structured protocol results without exposing OPY AST/HIR or Rust implementation types. diff --git a/docs/adr/0004-typed-semantic-feature-locality.md b/docs/adr/0004-typed-semantic-feature-locality.md index 868e1083..eaa10dc1 100644 --- a/docs/adr/0004-typed-semantic-feature-locality.md +++ b/docs/adr/0004-typed-semantic-feature-locality.md @@ -2,7 +2,7 @@ - Status: Accepted - Date: 2026-09-12 (backfilled) -- Historical evidence: [Issue #202](https://github.com/wrightkit/opy-rs/issues/202), [Issue #203](https://github.com/wrightkit/opy-rs/issues/203), [Issue #207](https://github.com/wrightkit/opy-rs/issues/207), [Issue #209](https://github.com/wrightkit/opy-rs/issues/209), [Issue #221](https://github.com/wrightkit/opy-rs/issues/221), [PR #205](https://github.com/wrightkit/opy-rs/pull/205), [PR #210](https://github.com/wrightkit/opy-rs/pull/210), [PR #213](https://github.com/wrightkit/opy-rs/pull/213), [PR #214](https://github.com/wrightkit/opy-rs/pull/214), [PR #222](https://github.com/wrightkit/opy-rs/pull/222), [current language-core contract](../architecture/language-core.md) +- Related history: [Issue #202](https://github.com/wrightkit/opy-rs/issues/202), [Issue #203](https://github.com/wrightkit/opy-rs/issues/203), [Issue #207](https://github.com/wrightkit/opy-rs/issues/207), [Issue #209](https://github.com/wrightkit/opy-rs/issues/209), [Issue #221](https://github.com/wrightkit/opy-rs/issues/221), [PR #205](https://github.com/wrightkit/opy-rs/pull/205), [PR #210](https://github.com/wrightkit/opy-rs/pull/210), [PR #213](https://github.com/wrightkit/opy-rs/pull/213), [PR #214](https://github.com/wrightkit/opy-rs/pull/214), [PR #222](https://github.com/wrightkit/opy-rs/pull/222), [current language-core contract](../architecture/language-core.md) ## Context @@ -21,7 +21,7 @@ rules, contextual dispatch, argument semantics, or special lowering. Implement observable OverPy behavior and invariants in typed Rust close to the owning language feature. Use validated or generated data for declarative facts such as names, aliases that are identities, signatures, enum membership, -catalog links, and provenance. Existing metadata-driven behavior is treated as +catalog links, and source attribution. Existing metadata-driven behavior is treated as architecture debt to audit, not as a reason to add more semantic control fields. @@ -39,7 +39,7 @@ where the existing contract requires it. - Keep all behavior in phase-wide modules: it minimizes file movement but makes unrelated features harder to locate and change safely. - Translate upstream module structure directly: it may resemble the oracle but - confuses compatibility evidence with an implementation boundary. + confuses compatibility tests with an implementation boundary. The chosen approach costs explicit Rust code and bounded internal module organization, while keeping behavior reviewable, typed, and near its owner. @@ -47,7 +47,7 @@ organization, while keeping behavior reviewable, typed, and near its owner. ## Consequences New language behavior should be placed by semantic responsibility and tested -through observable source, diagnostics, provenance, lowering, or tooling -contracts. Inventory completeness and conformance evidence remain important, +through observable source, diagnostics, source mapping, lowering, or tooling +contracts. Inventory completeness and compatibility tests remain important, but neither narrows the upstream language scope nor authorizes implementation behavior by itself. diff --git a/docs/adr/0005-opy-canonical-program-lowering.md b/docs/adr/0005-opy-canonical-program-lowering.md index 82769c40..0927399e 100644 --- a/docs/adr/0005-opy-canonical-program-lowering.md +++ b/docs/adr/0005-opy-canonical-program-lowering.md @@ -2,7 +2,7 @@ - Status: Accepted - Date: 2026-09-12 (backfilled) -- Historical evidence: [OPY Issue #244](https://github.com/wrightkit/opy-rs/issues/244), [OPY PR #248](https://github.com/wrightkit/opy-rs/pull/248), [Workshop Issue #179](https://github.com/wrightkit/workshop-rs/issues/179), [Workshop PR #185](https://github.com/wrightkit/workshop-rs/pull/185) +- Related history: [OPY Issue #244](https://github.com/wrightkit/opy-rs/issues/244), [OPY PR #248](https://github.com/wrightkit/opy-rs/pull/248), [Workshop Issue #179](https://github.com/wrightkit/workshop-rs/issues/179), [Workshop PR #185](https://github.com/wrightkit/workshop-rs/pull/185) - Owner decision: [workshop-rs ADR-0008](https://github.com/wrightkit/workshop-rs/blob/main/docs/adr/0008-canonical-public-program-boundary.md) ## Context @@ -21,10 +21,10 @@ and settings concepts through the public owner API. OPY must not recreate a local WIR/storage/node-ID model and convert it afterward. OPY retains ownership of helper/index allocation, preprocessing/macros, -source-language diagnostics, and provenance mapping. Where source provenance -is needed, OPY consumes the optional public provenance contract supplied by +source-language diagnostics, and source mapping. Where source mapping +is needed, OPY consumes the optional public source-mapping contract supplied by `workshop-rs`. Workshop owns canonical semantics, validation, catalog, -localization/settings, and emission. Compatibility evidence may compare +localization/settings, and emission. Compatibility tests may compare canonical semantics, but it does not make Workshop's internal WIR/storage a public OPY dependency. @@ -32,14 +32,14 @@ public OPY dependency. - Keep a local OPY WIR mirror and adapt later: it appears to isolate migration risk but duplicates canonical representation and can silently drop semantics - or provenance. +or source mapping. - Move OPY lowering policy into `workshop-rs`: it simplifies one call site but violates source-language ownership and couples Workshop to OPY. - Depend directly on Workshop storage APIs: it may expose more mechanics but freezes internal allocation and node identity as a consumer contract. The chosen boundary makes the Workshop owner API the integration seam while -keeping source-language decisions local and preserving direct provenance paths. +keeping source-language decisions local and preserving direct source mappings. ## Consequences diff --git a/docs/adr/0006-tests-first-compatibility.md b/docs/adr/0006-tests-first-compatibility.md new file mode 100644 index 00000000..1c5ed9ab --- /dev/null +++ b/docs/adr/0006-tests-first-compatibility.md @@ -0,0 +1,40 @@ +# ADR-0006: Tests-first compatibility verification + +- Status: Accepted +- Date: 2026-09-20 +- Related history: [opy-rs Issue #337](https://github.com/wrightkit/opy-rs/issues/337), [wrightkit/.github Issue #60](https://github.com/wrightkit/.github/issues/60) +- Supersedes: [ADR-0002](0002-independent-conformance-evidence.md) + +## Context + +OPY compatibility is a behavior requirement. The repository already has +ordinary source, compiler, regression, differential, and real-project tests, +along with pinned OverPy results and canonical Workshop comparisons. Separate +expectation tables and generic verification metadata duplicate those tests and +make the test contract harder to find. + +## Decision + +Each compatibility fixture owns its test input, concrete source attribution or +licensing data when required, pinned oracle snapshot, and the native +expectations needed by its source and compiler tests. The source and compiler +runners read those declarations directly from the fixture manifest. + +OverPy remains the pinned reference implementation. Its snapshots are test +data; canonical Workshop comparisons, diagnostic-code checks, normalized +output checks, and source status comparisons remain ordinary test assertions. +Generated reports are run artifacts under `target/` or CI artifacts and are +not current test metadata. + +Source mapping remains part of diagnostics and HIR where consumers need it. +Third-party source attribution, licensing, immutable revisions, checksums, and +reproducibility details remain only where the corresponding workflow needs +them. + +## Consequences + +Contributors can understand a compatibility check as fixture input plus native +execution, pinned reference result, and assertion. There is no separate +evidence manifest, generic provenance schema, expectation owner field, or +task-history table to maintain. The pinned oracle and canonical Workshop +ownership remain unchanged. diff --git a/docs/adr/README.md b/docs/adr/README.md index e44d7410..5978cb67 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -21,10 +21,11 @@ that later implementation still conforms to it. ## Index - [ADR-0001: Public Rust embedding surface](0001-public-rust-embedding-surface.md) -- [ADR-0002: Independent offline OverPy conformance evidence](0002-independent-conformance-evidence.md) +- [ADR-0002: Independent offline OverPy conformance evidence](0002-independent-conformance-evidence.md) (Superseded) - [ADR-0003: First-party provider ownership boundary](0003-first-party-provider-boundary.md) - [ADR-0004: Typed semantic behavior and feature-local ownership](0004-typed-semantic-feature-locality.md) - [ADR-0005: OPY lowering through canonical Workshop `Program`](0005-opy-canonical-program-lowering.md) +- [ADR-0006: Tests-first compatibility verification](0006-tests-first-compatibility.md) ## Post-baseline audit (#250) @@ -36,7 +37,7 @@ current contracts, implementation detail, and decisions owned elsewhere. | Historical choice | Classification | Decision record | | --- | --- | --- | | #111 and PR #112 | Backfill required: intentional public Rust package and embedding boundary | [ADR-0001](0001-public-rust-embedding-surface.md) | -| #158 and PR #159 | Backfill required: independent oracle, stage-aware failure frontier, and canonical semantic comparison | [ADR-0002](0002-independent-conformance-evidence.md) | +| #158 and PR #159 | Historical choice: independent oracle, stage-aware failure frontier, and canonical semantic comparison | [ADR-0002](0002-independent-conformance-evidence.md) | | #170 and PR #171 | Backfill required: thin first-party provider over OPY-owned loading and semantics | [ADR-0003](0003-first-party-provider-boundary.md) | | Current `language-core.md`, #202, #203/#205, #207/#210, #208/#213, #209/#214, and #221/#222 | Backfill required: upstream executable specification, typed behavior, declarative facts, and domain-local ownership | [ADR-0004](0004-typed-semantic-feature-locality.md) | | #244/#248 and the Workshop owner decision | Backfill required for OPY consequences; canonical `Program` representation is externally owned | [ADR-0005](0005-opy-canonical-program-lowering.md), [workshop-rs ADR-0008](https://github.com/wrightkit/workshop-rs/blob/main/docs/adr/0008-canonical-public-program-boundary.md) | @@ -50,5 +51,5 @@ this repository records only how OPY consumes that boundary. No unresolved OPY architecture decision was identified in this audit. The ADRs preserve recoverable rationale; current architecture documents remain the -authority for present boundaries, and implementation evidence must still be +authority for present boundaries, and current implementation reality must still be checked separately. diff --git a/docs/architecture/README.md b/docs/architecture/README.md index e0918968..ccfcbba3 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -2,11 +2,11 @@ This directory routes the **current** architecture contracts for `opy-rs`. -Use it for substantive implementation preflight. Architecture intent, implementation reality, compatibility evidence, and historical design notes are separate: +Use it for substantive implementation preflight. Architecture intent, implementation reality, compatibility tests, and historical design notes are separate: - documents here state the current durable language/ownership contracts; -- source, Cargo metadata, tests, corpus, differential evidence, and real projects establish implementation reality; -- `docs/language-support.md` records current evidenced support, not the definition of what the OverPy core language is; +- source, Cargo metadata, tests, corpus, reference comparisons, and real projects establish implementation reality; +- `docs/language-support.md` records current tested support, not the definition of what the OverPy core language is; - older implementation documents under `docs/opy/` describe prior/current mechanisms but are not semantic authority merely because code already uses them. ## Routing @@ -15,14 +15,14 @@ Use it for substantive implementation preflight. Architecture intent, implementa | --- | --- | | OverPy language ownership, upstream core scope, semantic implementation model | [`language-core.md`](language-core.md) | | Boundary with canonical Workshop and source→Workshop / Workshop→source responsibilities | [`workshop-boundary.md`](workshop-boundary.md) | -| Current evidenced support | [`../language-support.md`](../language-support.md) plus executable evidence | -| Pinned upstream identity and provenance | [`../compatibility/upstream-references.md`](../compatibility/upstream-references.md) | +| Current tested support | [`../language-support.md`](../language-support.md) plus executable tests | +| Pinned upstream identity and source attribution | [`../compatibility/upstream-references.md`](../compatibility/upstream-references.md) | | Opy HIR wire/semantic representation | [`../hir/opy-hir-v2.md`](../hir/opy-hir-v2.md) where still consistent with current contracts and code reality | | Tooling/API contract | [`../opy/tooling-api.md`](../opy/tooling-api.md) | | Provider integration | [`../opy/provider.md`](../opy/provider.md) | Historical rationale for material decisions is recorded in the [ADR registry](../adr/README.md). ADRs do not replace these current contracts -or implementation evidence. +or implementation test results. Do not put release versions, feature counts, current Issue progress, or transient compiler gaps in this directory. If an Issue or older design document conflicts with these contracts or current code reality, surface the mismatch before implementation. diff --git a/docs/architecture/language-core.md b/docs/architecture/language-core.md index 87506cd6..c41e5b71 100644 --- a/docs/architecture/language-core.md +++ b/docs/architecture/language-core.md @@ -18,7 +18,7 @@ Upstream implementation structure is not an architecture mandate. `opy-rs` shoul - preprocessing, includes, defines, macros, and source directives; - source-language name/member/signature resolution; - OverPy type and contextual semantics; -- source diagnostics and provenance; +- source diagnostics and source mapping; - OPY HIR and source-aware tooling semantics; - OverPy-specific lowering/compiler behavior; - Workshop→OPY reconstruction. @@ -36,7 +36,7 @@ Implement program behavior and semantic invariants in typed Rust. Examples inclu - special lowering decisions; - compile-time evaluation and control flow. -Validated/generated data is appropriate for large declarative inventories such as names, aliases that are purely identities, mechanical signatures, provenance links, and completeness inventories when those records do not themselves become a language for programming semantics. +Validated/generated data is appropriate for large declarative inventories such as names, aliases that are purely identities, mechanical signatures, source attribution links, and completeness inventories when those records do not themselves become a language for programming semantics. A manifest field that causes generic Rust to choose source-language behavior is not justified merely because an existing manifest already contains similar fields. When metadata drives receiver restrictions, argument semantics, contextual dispatch, special lowering, or other observable behavior, treat the placement as architecture debt to audit and prefer direct typed implementation unless a concrete domain reason justifies a declarative representation. @@ -59,11 +59,11 @@ Compatibility work should preserve, where the canonical Workshop model can repre - upstream lowering choices when an alternative form adds no value; - upstream string/value construction shape when divergence only adds Workshop elements; - optimizer behavior that materially affects Workshop element cost or emitted structure; -- stable source-to-output behavior demonstrated by differential or real-project evidence. +- stable source-to-output behavior demonstrated by differential or real-project tests. A systematic output difference is a compatibility residual by default. It should be explained and classified rather than dismissed merely because the resulting program may appear behaviorally equivalent. -Exact byte-for-byte identity is not a universal completion requirement. Incidental whitespace, formatting, or other representation details may differ when they do not change Workshop structure, element cost, accepted syntax, diagnostics/provenance, or downstream behavior. Likewise, upstream internal architecture, helper names, and IR remain non-contractual. +Exact byte-for-byte identity is not a universal completion requirement. Incidental whitespace, formatting, or other representation details may differ when they do not change Workshop structure, element cost, accepted syntax, source mapping, or downstream behavior. Likewise, upstream internal architecture, helper names, and IR remain non-contractual. The practical objective is to make supported `opy-rs` compilation converge as closely as reasonably possible on upstream Workshop output while retaining WrightKit's canonical ownership boundaries. Where equivalent canonical representations exist, prefer the representation demonstrated by upstream unless there is concrete evidence that divergence is necessary. diff --git a/docs/architecture/workshop-boundary.md b/docs/architecture/workshop-boundary.md index aa6a6c97..2bd1c5b0 100644 --- a/docs/architecture/workshop-boundary.md +++ b/docs/architecture/workshop-boundary.md @@ -41,7 +41,7 @@ emission. Its arena-backed WIR/storage is internal support representation. The ordinary consumer boundary is the public `workshop-rs::Program` model; arena-backed WIR/storage and node-ID mechanics are internal to `workshop-rs`, as recorded by [workshop-rs ADR-0008](https://github.com/wrightkit/workshop-rs/blob/main/docs/adr/0008-canonical-public-program-boundary.md). -OPY may use canonical semantic APIs and approved provenance access, but must +OPY may use canonical semantic APIs and approved source-mapping access, but must not recreate or depend on that internal storage representation. Do not move OverPy names, aliases, contextual dispatch records, compiler helper identities, or reconstruction carriers into canonical Workshop merely to simplify compilation. diff --git a/docs/benchmarks/compiler-resources-baseline.json b/docs/benchmarks/compiler-resources-baseline.json index cc431348..b7cd7dee 100644 --- a/docs/benchmarks/compiler-resources-baseline.json +++ b/docs/benchmarks/compiler-resources-baseline.json @@ -19,7 +19,7 @@ "parameters": { "depth": 24 }, - "provenance": { + "attribution": { "kind": "original", "license": "AGPL-3.0-or-later", "origin": "WrightKit opy-rs resource audit workload", @@ -100,7 +100,7 @@ "parameters": { "vectorCount": 32 }, - "provenance": { + "attribution": { "kind": "original", "license": "AGPL-3.0-or-later", "origin": "WrightKit opy-rs resource audit workload", @@ -181,7 +181,7 @@ "parameters": { "ruleCount": 128 }, - "provenance": { + "attribution": { "kind": "original", "license": "AGPL-3.0-or-later", "origin": "WrightKit opy-rs resource audit workload", @@ -262,7 +262,7 @@ "parameters": { "constructions": 64 }, - "provenance": { + "attribution": { "kind": "original", "license": "AGPL-3.0-or-later", "origin": "WrightKit opy-rs resource audit workload", @@ -343,7 +343,7 @@ "parameters": { "invocations": 64 }, - "provenance": { + "attribution": { "kind": "original", "license": "AGPL-3.0-or-later", "origin": "WrightKit opy-rs resource audit workload", @@ -424,7 +424,7 @@ "parameters": { "settingsKeyCount": 1000 }, - "provenance": { + "attribution": { "kind": "original", "license": "AGPL-3.0-or-later", "origin": "WrightKit opy-rs resource audit workload", @@ -498,16 +498,16 @@ "workload": "large-settings-source" }, { - "description": "provenance-linked OverPy parabola example compiled end to end", + "description": "attributed OverPy parabola example compiled end to end", "input": { "bytes": 3035, - "kind": "provenance-linked-corpus", + "kind": "attributed-corpus", "parameters": { "entry": "parabola.opy", "fixture": "real-world/overpy-parabola", "fixtureMetadata": "crates/opy-rs/tests/fixtures/corpus/real-world/overpy-parabola/fixture.json" }, - "provenance": { + "attribution": { "kind": "imported-example", "license": "GPL-3.0-only", "licenseUrl": "https://github.com/Zezombye/overpy/blob/eea67adbcf6926c4004e35e25ab4be072624a44e/LICENSE", diff --git a/docs/benchmarks/compiler-resources.md b/docs/benchmarks/compiler-resources.md index e7aae7bf..bda12d72 100644 --- a/docs/benchmarks/compiler-resources.md +++ b/docs/benchmarks/compiler-resources.md @@ -4,7 +4,7 @@ This is the reproducible resource baseline for [opy-rs#296](https://github.com/w The accepted baseline record is [`compiler-resources-baseline.json`](compiler-resources-baseline.json), generated from the audited `origin/main` revision on macOS arm64 with five samples per workload. It records seven independent workloads; the instrumented candidate workflow adds the `macro-runtime-heavy` workload: -| Workload | Mechanism isolated | Input and provenance | +| Workload | Mechanism isolated | Input and attribution | | --- | --- | --- | | `nested-expression-lowering` | Nested lowering values and subtree copies | Generated source, depth 24, SHA-256 recorded in the baseline record | | `wide-call-value-construction` | Wide value construction and nested argument copies | Generated `[vect(...)]`, 32 vectors, SHA-256 recorded in the baseline record | @@ -13,7 +13,7 @@ The accepted baseline record is [`compiler-resources-baseline.json`](compiler-re | `macro-runtime` | QuickJS engine creation per invocation | Fixed script `(x + 2).toString();`, 64 invocations, SHA-256 recorded in the baseline record | | `macro-runtime-heavy` | QuickJS lifecycle cost with non-trivial script work | Helper- and collection-heavy script, 64 invocations, SHA-256 recorded in the candidate record | | `large-settings-source` | Settings scanner materialization | Generated JSONC-like block with 1,000 keys, SHA-256 recorded in the baseline record | -| `real-world-parabola` | End-to-end compiler path on a provenance-linked project | `tests/fixtures/corpus/real-world/overpy-parabola`, including its `fixture.json` provenance and source SHA-256 | +| `real-world-parabola` | End-to-end compiler path on an attributed project | `tests/fixtures/corpus/real-world/overpy-parabola`, including its `fixture.json` attribution and source SHA-256 | The test-only counters report retained lowering values, copied value-tree nodes from the pre-#299 lowering path, canonical value materialization nodes, action-copy events, manifest/catalog contract checks, materialized settings characters, and QuickJS engine creations. The clone counters retain their original #296 meanings so before/after and ablation records remain comparable; they are zero for the #299 candidate because the copied lowering path has been removed. For the two macro workloads they also report aggregate nanoseconds for runtime/context creation, host registration, builtin-helper evaluation, user-script evaluation, and runtime/context teardown. They are compiled only for unit tests and do not add a production API or telemetry surface. Each workload runs in a separate child test process; peak RSS is read with `getrusage(RUSAGE_SELF)` inside that child. macOS reports bytes directly; Linux converts the kernel's KiB value to bytes. Other platforms report `null` for RSS. @@ -35,7 +35,7 @@ To intentionally refresh the pinned record, run the command at the audited defau For a before/after comparison, run the command once at the audited `origin/main` revision and once at the candidate revision, keeping OS, architecture, Rust version, build profile, and workload parameters unchanged. Compare elapsed time directionally and compare mechanism counters exactly; do not turn a single-machine timing into a fixed percentage contract. Repeat the command when a directional result matters and report run count and environmental noise with the result. -The real-project case is intentionally the small, successful `overpy-parabola` fixture so a complete lowering/emission path is exercised without making a large corpus failure the benchmark's success condition. Its committed `fixture.json`, source hash, upstream commit, license, and source URL remain the provenance authority. +The real-project case is intentionally the small, successful `overpy-parabola` fixture so a complete lowering/emission path is exercised without making a large corpus failure the benchmark's success condition. Its committed `fixture.json`, source hash, upstream commit, license, and source URL remain the attribution authority. ## QuickJS lifecycle conclusion (#300) diff --git a/docs/compatibility/upstream-references.md b/docs/compatibility/upstream-references.md index eaef90ae..2150e91c 100644 --- a/docs/compatibility/upstream-references.md +++ b/docs/compatibility/upstream-references.md @@ -1,11 +1,11 @@ -# Upstream Reference and Provenance: OverPy +# Upstream Reference and Licensing: OverPy Status: accepted baseline (issue #2). The pinned compatibility oracle and the -clean-room/provenance policy for the opy-rs evidence base -Scope: project-level provenance for the OverPy reference `opy-rs` studies or -derives compatibility knowledge from; the durable record that lets `opy-rs` -read and reference upstream source without per-symbol or per-file provenance -bureaucracy +clean-room/source-attribution policy for OPY compatibility tests. +Scope: project-level identity, licensing, and attribution for the OverPy +reference `opy-rs` studies or derives compatibility behavior from; the durable +record that lets `opy-rs` read and reference upstream source without per-symbol +or per-file attribution bureaucracy. `opy-rs` is independently implemented Rust software (AGPL-3.0-or-later) that reimplements OverPy source-language semantics. Reimplementing compatible @@ -33,7 +33,7 @@ not legal advice and does not settle questions that require a qualified lawyer. | Registry integrity | `sha512-oX17nauJcPTaKIrRFY/rD0Rl8atqFUVv9Hg2TKH+A68/fC8+ZO344Mkd1A/Y0oOVp1hr5tktMBjzMEDDnMEYUw==` (recorded in `tools/overpy/oracle/oracle-metadata.json` and the lockfile) | | Recorded `gitHead` | `1e2688954302a402d076944b46db07efb14d7b61`. npm's `gitHead` field lags the tarball content by one release; it is the `v9.7.9` tag commit and must **not** be treated as the content commit | | License assumption | GPL-3.0-only (engineering assumption, not a legal conclusion; the npm `package.json` ships no `license` field, see `tools/overpy/oracle/oracle-metadata.json`) | -| Language | en-US (Workshop locale for reference evidence) | +| Language | en-US (Workshop locale for reference tests) | The integrity hash pins the content. Reproduction uses the recorded identity, never `latest` or a range (see the pinning policy below). @@ -64,7 +64,7 @@ dependency of `opy-rs` and is never bundled into release artifacts. Concretely, it serves as: * the reference for S (syntax), D (diagnostic), and N (normalized-output) - evidence in the compatibility corpus (`crates/opy-rs/tests/fixtures/corpus/**`, + tests in the compatibility corpus (`crates/opy-rs/tests/fixtures/corpus/**`, `tools/overpy/oracle/`); * the independent reference for support claims in [`docs/language-support.md`](../language-support.md), exercised through the @@ -73,7 +73,7 @@ it serves as: ([`docs/hir/opy-hir-v2.md`](../hir/opy-hir-v2.md)): the native differential suite (`crates/opy-rs/tests/differential.rs`, merged in PR #13) runs every corpus fixture through the native pipeline in `cargo test` and - compares status, rule-name, and diagnostic evidence against the recorded + compares status, rule names, and diagnostics against the recorded oracle snapshots. ### Invocation records @@ -129,7 +129,7 @@ grounded in the pinned tree, specifically: opy-rs-owned semantic manifest records only oracle-validated facts (see `compat-manifest-spec.md`). -## Clean-room and provenance policy +## Clean-room and source-attribution policy The policy below is the opy-rs-adapted summary of WrightKit's accepted component boundary (wright `docs/licensing.md`) and reference pinning policy @@ -141,8 +141,8 @@ component boundary (wright `docs/licensing.md`) and reference pinning policy | --- | --- | --- | | `opy-rs` core (lexer, preprocess, CST/parser, semantic resolution, HIR, diagnostics) | No | Independently implemented code. It must not link to the reference, copy its source, import its internal AST/types, or compile against its generated artifacts. | | Compatibility harness / oracle tool | Yes, for isolated evaluation | It may invoke a separately installed/pinned reference and compare documented or generated results. It must remain separable from the core build and runtime distribution. | -| Compatibility fixtures (upstream example/test corpus) | Only after provenance review | Provenance/license/redistribution-reviewed upstream example and test fixture files (e.g. the GPL-3.0 OverPy `examples/*.opy` corpus) may be retained under `crates/opy-rs/tests/fixtures/corpus/` as documented, isolated oracle evidence, with per-file origin, license, redistribution status, byte-identity against the pinned content commit, and SHA-256 records (see the fixture corpus policy below). They are never imported by core code and never bundled into core builds or release artifacts. | -| Generated reference artifacts (oracle snapshots, manifests) | Only after provenance review | Store identifiers, hashes, generators, or reviewable artifacts only when their license and redistribution status are recorded. Do not add reference implementation/data content or unclear third-party content. | +| Compatibility fixtures (upstream example/test corpus) | Only after attribution review | Attributed, licensed, redistribution-reviewed upstream example and test fixture files (e.g. the GPL-3.0 OverPy `examples/*.opy` corpus) may be retained under `crates/opy-rs/tests/fixtures/corpus/` as ordinary test inputs, with per-file origin, license, redistribution status, byte identity against the pinned content commit, and SHA-256 records (see the fixture corpus policy below). They are never imported by core code and never bundled into core builds or release artifacts. | +| Generated reference artifacts (oracle snapshots, manifests) | Only after attribution review | Store identifiers, hashes, generators, or reviewable artifacts only when their license and redistribution status are recorded. Do not add reference implementation/data content or unclear third-party content. | | CI and development scripts | Yes, when isolated | They may install or invoke a pinned external oracle for a compatibility check, but must not silently turn it into a core dependency or bundled release component. | No allow-listed path may import reference implementation details into the @@ -152,16 +152,16 @@ or README and linked from this document before it is used. ### Fixture corpus policy -`crates/opy-rs/tests/fixtures/corpus/` may retain provenance/license/redistribution-reviewed +`crates/opy-rs/tests/fixtures/corpus/` may retain attribution/license/redistribution-reviewed upstream example and test fixture files, e.g. the GPL-3.0 OverPy -`examples/*.opy` corpus, as documented, isolated oracle evidence. Each +`examples/*.opy` corpus, as documented test inputs. Each imported file carries its per-file record (origin, license, redistribution status, byte-identity against the pinned content commit, SHA-256) in `crates/opy-rs/tests/fixtures/corpus/README.md` and its `fixture.json`; that record is -authoritative and is not duplicated here. The fixture corpus is oracle -evidence, not a core input: core code never imports it, and it is never +authoritative and is not duplicated here. The fixture corpus is test data, +not a core input: core code never imports it, and it is never bundled into core builds or release artifacts. Content with unclear -provenance or no license is prohibited, and OverPy implementation or data +attribution or no license is prohibited, and OverPy implementation or data (`src/` sources, `src/data/*` tables, internal AST/types, generated artifacts) must not be imported into the core. @@ -172,7 +172,7 @@ The core may be developed from: * independently authored `opy-rs` code; * public language or output specifications, subject to their own license; * behavior observed through lawful, documented compatibility tests; -* a separately specified interchange format whose provenance and license are +* a separately specified interchange format whose attribution and license are known; and * third-party dependencies whose license and compatibility have been reviewed. @@ -186,10 +186,10 @@ Contributors working on the core must: 1. implement opy-rs-owned data structures and transformations rather than mechanically translating reference source or types; -2. keep source provenance for imported examples, fixtures, and generated +2. keep source attribution for imported examples, fixtures, and generated artifacts; 3. record the reference version and acquisition method for compatibility - evidence; and + tests; and 4. stop and request review when a proposed dependency, fixture, or code sample has unclear licensing or would place a reference implementation detail in a core API. @@ -210,7 +210,7 @@ on **demonstrated behavioral need**, never on release recency: and the byte-verified git content commit. A version bump alone is not an oracle change. 3. **Demonstrated need only.** "Demonstrated" means a version-sensitivity run - (the minimal repro plus the evidence source against candidate versions) + (the minimal repro plus the comparison result against candidate versions) showing a different accept/reject outcome or a different normalized output for a construct the corpus needs. Absence of measured divergence is a no-change decision. @@ -223,7 +223,7 @@ on **demonstrated behavioral need**, never on release recency: re-baseline. A pin change follows the structured review path: `oracle-metadata.json`, -lockfile, `run_oracle.py --update` snapshot review, fixture provenance notes, +lockfile, `run_oracle.py --update` snapshot review, fixture attribution notes, and the affected docs in one reviewed change. ### Distribution policy @@ -242,7 +242,7 @@ when the oracle is absent. changed only on demonstrated behavioral need, never on release recency. A version bump alone is not an oracle change. * **Measured stability.** Every accept/reject outcome and diagnostic in the - ported evidence set is byte-identical across `9.7.10 → 9.7.13` (measured in + ported fixture set is byte-identical across `9.7.10 → 9.7.13` (measured in the WrightKit Track B investigation); only hero/settings schema data differs. Historical claims stay interpretable because every result records the exact pinned identity. @@ -263,5 +263,5 @@ when the oracle is absent. * [`docs/language-support.md`](../language-support.md): public support contract and current states * [`docs/opy/tooling-notes.md`](../opy/tooling-notes.md): harness usage * [`tools/overpy/README.md`](../../tools/overpy/README.md): oracle and fixture layout -* [`crates/opy-rs/tests/fixtures/corpus/README.md`](../../crates/opy-rs/tests/fixtures/corpus/README.md): corpus provenance +* [`crates/opy-rs/tests/fixtures/corpus/README.md`](../../crates/opy-rs/tests/fixtures/corpus/README.md): fixture attribution and layout * WrightKit's policy sources this document adapts: `wright/docs/licensing.md`, wright ADR-0004 (OverPy licensing and clean-room boundary), ADR-0007 (reference pinning policy) diff --git a/docs/language-support.md b/docs/language-support.md index ef17a6ee..ba7ef260 100644 --- a/docs/language-support.md +++ b/docs/language-support.md @@ -36,20 +36,19 @@ supported subset and limitation are stated in user terms. | OPY → Workshop compilation | 🚧 Partial | [Tooling and backend](language-support/tooling-and-backend.md) | | Workshop → OPY reconstruction | ❌ Unsupported | [Tooling and backend](language-support/tooling-and-backend.md) | -## Evidence and ownership +## Tests and ownership -Support claims are grounded in executable evidence rather than a parallel -support database: +Support claims are grounded in executable tests and reference comparisons: -- the corpus under `crates/opy-rs/tests/fixtures/corpus/` preserves - provenance-linked projects and minimized regressions; -- `tools/overpy/oracle/` and each fixture's `oracle.json` record pinned upstream - behavior; +- the corpus under `crates/opy-rs/tests/fixtures/corpus/` contains source, + compiler, regression, and real-project test inputs; +- `tools/overpy/oracle/` and each fixture's `oracle.json` provide pinned + upstream reference results; - `tools/overpy/run_native.py` and `tools/overpy/diff.py` compare native output, - diagnostics, normalized Workshop output, and canonical-WIR evidence; + diagnostics, normalized Workshop output, and canonical-WIR comparisons; - Rust integration tests exercise source semantics and canonical lowering; - `cargo test -p opy-rs --test differential` checks the native source pipeline - against the corpus expectations. + against the fixture manifests and pinned snapshots. `opy-rs` owns OverPy syntax, preprocessing, source semantics, diagnostics and OverPy-specific lowering. `workshop-rs` owns canonical Workshop identities, diff --git a/docs/opy/compat-manifest-spec.md b/docs/opy/compat-manifest-spec.md index 939b86b2..9e587cdd 100644 --- a/docs/opy/compat-manifest-spec.md +++ b/docs/opy/compat-manifest-spec.md @@ -9,7 +9,7 @@ Current architecture is defined by [`docs/architecture/language-core.md`](../arc The manifest under `crates/opy-rs/src/manifest/` carries declarative compatibility data: - identities, names, aliases, signatures (including parameter defaults, optionality, - binding spellings, and enum-domain links), catalog links, and provenance. + binding spellings, and enum-domain links), catalog links, and source attribution. Typed feature-local lowering policy owns behavioral contextual dispatch and call-context restrictions: currently `chase` selector dispatch and `range`'s for-iterable-only rule @@ -29,8 +29,11 @@ OverPy-specific names, aliases, special forms, contextual behavior, and compiler See [`docs/architecture/workshop-boundary.md`](../architecture/workshop-boundary.md). -## Evidence and provenance +## Reference tests and attribution -Pinned upstream identity and licensing/provenance rules remain documented in [`docs/compatibility/upstream-references.md`](../compatibility/upstream-references.md). Manifest probes and differential tests remain useful compatibility evidence; they do not re-authorize or redefine established core features. +Pinned upstream identity, source attribution, and licensing rules remain +documented in [`docs/compatibility/upstream-references.md`](../compatibility/upstream-references.md). +Manifest probes and differential tests are ordinary compatibility tests; they +do not re-authorize or redefine established core features. The prior detailed manifest schema and field-by-field rationale are preserved in Git history. They describe how the current implementation evolved, not the current language architecture contract. diff --git a/docs/opy/implementation-role.md b/docs/opy/implementation-role.md index 969eac2a..c7783cbf 100644 --- a/docs/opy/implementation-role.md +++ b/docs/opy/implementation-role.md @@ -8,4 +8,4 @@ mutable support state or new architecture decisions here. Use: - [`language-core.md`](../architecture/language-core.md) for OverPy language ownership and semantic implementation rules; - [`workshop-boundary.md`](../architecture/workshop-boundary.md) for the canonical Workshop dependency boundary; -- [`../language-support.md`](../language-support.md) plus executable evidence for current support reality. +- [`../language-support.md`](../language-support.md) plus executable tests for current support reality. diff --git a/docs/opy/tooling-api.md b/docs/opy/tooling-api.md index 2c228314..52280cf3 100644 --- a/docs/opy/tooling-api.md +++ b/docs/opy/tooling-api.md @@ -35,21 +35,18 @@ frontend class; canonical Workshop, locale, directive, and hook failures use the integration class. Normalized output removes line-ending and trailing presentation noise, while exact output preserves the emitted artifact. -Compatibility evidence is produced by the isolated compatibility harness, not -by the ordinary compile API. For semantic-WIR cases, the feature-gated internal -`opy-compat` target parses only the pinned reference Workshop text and compares -it directly with the native lowered WIR through +Compatibility comparisons are produced by the isolated compatibility harness, +not by the ordinary compile API. For semantic-WIR cases, the feature-gated +internal `opy-compat` target parses only the pinned reference Workshop text and +compares it directly with the native lowered WIR through `workshop-rs::roundtrip::equivalent`. It writes harness-level -`compatibility.semanticWIR` evidence, including the project and reference -input digests; the public compile report and `opy-cli compile` have no oracle -input or compatibility-evidence field. - -The compatibility runner uses the separate reviewed compiler expectation -contract in -[`tools/overpy/compiler-expectations.json`](../../tools/overpy/compiler-expectations.json) -for compiler outcomes. The source/frontend expectation contract is -kept in `differential-expectations.json`; it is not reused as compiler parity -evidence. Compiler gaps must carry durable evidence and an owner, while +`compatibility.semanticWIR` comparison data, including the project and +reference input digests; the public compile report and `opy-cli compile` have +no oracle input or comparison field. + +The compatibility runner reads source and compiler expectations from each +fixture's `fixture.json`. A compiler comparison can use normalized output, +semantic WIR, diagnostic codes, or an explicitly bounded compiler contract; expectation mismatches remain blocking. ## Library API (`opy_rs::tooling`) diff --git a/docs/opy/tooling-notes.md b/docs/opy/tooling-notes.md index 24641127..0d28e27e 100644 --- a/docs/opy/tooling-notes.md +++ b/docs/opy/tooling-notes.md @@ -1,6 +1,6 @@ # opy-rs Compatibility Tooling Notes -Small operational notes for the OverPy evidence harness; the full layout +Small operational notes for the OverPy compatibility harness; the full layout contract is in [`tools/overpy/README.md`](../../tools/overpy/README.md). ## Prerequisites @@ -25,7 +25,7 @@ python3 tools/overpy/run_oracle.py --update # Verify snapshots still match the pinned oracle (fails on any mismatch) python3 tools/overpy/run_oracle.py -# Compiler compatibility gate (public CLI plus internal evidence target) +# Compiler compatibility gate (public CLI plus internal comparison target) cargo build --locked -p opy-cli --bin opy-cli cargo build --locked -p opy-cli --features compatibility --bin opy-compat python3 -B tools/overpy/run_native.py \ @@ -46,6 +46,6 @@ python3 tools/overpy/diff.py --producer-command '' --report target A pin change is an explicit, reviewed change: update `tools/overpy/oracle/package.json` + `pnpm-lock.yaml` + `oracle-metadata.json`, re-run `run_oracle.py --update`, review every snapshot -diff and fixture provenance note, and update the reference identity records in +diff and fixture attribution note, and update the reference identity records in `docs/compatibility/upstream-references.md` (policy: changed only on demonstrated behavioral need, never on release recency). diff --git a/docs/opy/trivia-retention-policy.md b/docs/opy/trivia-retention-policy.md index 795be7dd..9dcfee89 100644 --- a/docs/opy/trivia-retention-policy.md +++ b/docs/opy/trivia-retention-policy.md @@ -1,4 +1,4 @@ -# Trivia and Source-Provenance Retention Policy +# Trivia and Source-Mapping Retention Policy Status: accepted policy. Issue #3 acceptance. Scope: what the OPY source implementation retains from authored source and what it @@ -12,8 +12,8 @@ intentionally discards, for the Workshop-independent source implementation surfa | Line comments (`# …`) and block comments (`/* … */`) | **No** | The lexer discards comments before tokenization (they never enter the token stream) | | Whitespace and indentation | No (reconstructed deterministically) | The CST stores statements/blocks, not original indentation | | Source spans | Yes | 1-based line/column spans per token and CST node; `OpyError` diagnostics carry spans; the file registry maps span file ids to paths | -| File provenance | Yes | Preprocess `FileRecord` per file (id + path); HIR `SourceFile` entries; spans are attributed across include boundaries | -| Macro/define expansion provenance | Yes | `#!define` expansions carry the define's span; diagnostics attribute to authored and expansion sites | +| File mapping | Yes | Preprocess `FileRecord` per file (id + path); HIR `SourceFile` entries; spans are attributed across include boundaries | +| Macro/define expansion mapping | Yes | `#!define` expansions carry the define's span; diagnostics attribute to authored and expansion sites | | Settings blocks | No (consumed pre-lexing) | Parsed into the typed settings payload; source layout not retained | ## Rationale diff --git a/tools/overpy/README.md b/tools/overpy/README.md index 4ecea587..fe9d52ed 100644 --- a/tools/overpy/README.md +++ b/tools/overpy/README.md @@ -1,191 +1,60 @@ -# OverPy Evidence Harness +# OverPy compatibility test harness -This directory contains the reproducible OverPy evidence harness described by -[`docs/opy/tooling-notes.md`](../../docs/opy/tooling-notes.md). It is evaluation -tooling, not a dependency of the opy-rs core, and it is fully independent of -the Rust crates: no Node toolchain and no source implementation crate is required to run -the harness tests or the fixture snapshot checks. +This directory contains the pinned OverPy reference runner and the scripts +that execute ordinary OPY source and compiler differential tests. It is +evaluation tooling, not a dependency of the `opy-rs` core. ## Pinned oracle -`oracle/package.json` pins the OverPy npm package to `9.7.10`, while -`oracle/pnpm-lock.yaml` pins the resolved package integrity. The adjacent -`oracle-metadata.json` records the package's npm `gitHead`, repository, registry -tarball, license assumption, and Workshop language. A change to any of these is -an explicit oracle-version review (see -[`docs/compatibility/upstream-references.md`](../../docs/compatibility/upstream-references.md)). - -Install the oracle from the repository root: +`oracle/package.json` pins OverPy `9.7.10`; `oracle/pnpm-lock.yaml` pins its +integrity. `oracle-metadata.json` records the package identity, repository, +license assumption, and Workshop language. The oracle is installed separately +and is never bundled into `opy-rs` or imported by the Rust core. ```sh pnpm install --dir tools/overpy/oracle ``` -The oracle is not bundled into opy-rs and is not imported by the Rust core. +## Fixtures and tests -## Fixture layout +Each fixture under +`crates/opy-rs/tests/fixtures/corpus///` contains a +`fixture.json` manifest, OPY input, and `oracle.json` result snapshot. The +manifest is the only per-fixture expectation record. When needed, it keeps +concrete source attribution/licensing and reproducibility data, plus the source +and compiler test declarations. The snapshot records the pinned reference output +and the complete source-project digest. -Each fixture lives in its own directory: +The Rust differential test runs the native frontend without Node or OverPy: -```text -crates/opy-rs/tests/fixtures/corpus/// - fixture.json # metadata and expected compile status - source.opy # input, or the source path named by fixture.json - oracle.json # normalized result snapshot generated by the runner +```sh +cargo test -p opy-rs --test differential ``` -`fixture.json` schema version 1 requires: - -* `id`, unique across the corpus; -* `category`, used for aggregate reports; -* optional `features`, naming the syntax or semantic areas covered; -* `source`, relative to the fixture directory; -* `expectedStatus`, either `success` or `failure`; and -* `provenance` with `kind`, `origin`, `license`, and `redistributable`. - -Imported fixtures should also record an immutable `sourceCommit`, a direct -`sourceUrl`, a `licenseUrl`, and whether the source was modified. The corpus -contains WrightKit-authored synthetic cases, a census boundary fixture, and -provenance-recorded real-world projects derived from the pinned OverPy -`examples/` tree or independent BSD-2-Clause projects, with full include -closures. See `crates/opy-rs/tests/fixtures/corpus/README.md` for the complete provenance -record; the current inventory is the per-fixture metadata discovered and -validated by [`run_oracle.py`](run_oracle.py). - -`oracle.json` captures the pinned oracle identity, the complete resolved OPY -project input manifest (relative source paths and per-file SHA-256 hashes), its -canonical project digest, compile status, exit code, normalized diagnostics, -normalized Workshop text, and normalized output hash. The runner normalizes -line endings, trailing whitespace, and final newline presentation only; it does -not remove Workshop operations or values. The project manifest covers every -`.opy` file in the fixture directory except explicitly listed minimized -regression snippets, so changes to included sources invalidate the snapshot. - -[`crates/opy-rs/tests/differential-expectations.json`](../../crates/opy-rs/tests/differential-expectations.json) -is the independent native-side expectation record. Each fixture has a native -outcome, an expected relationship to the oracle (`match`, `known-gap`, or -`unsupported`), a rationale, and evidence. -The differential runner derives `unexpected-divergence`, `regression`, and -`inconclusive` results from those records; it never treats a reference-success -/ native-failure case as a match. - -`compiler-expectations.json` is a separate contract for the #38 compiler -surface. It does not replace or weaken the source/frontend expectations above: -its entries declare the compiler status, normalized-output/direct semantic-WIR -or diagnostic contract, evidence, and concrete owner for each fixture. A -non-match must cite both its pinned oracle snapshot and fixture provenance; -parent issue #8 is not a durable gap owner. - -For a `normalized-output` compiler comparison, non-numeric text remains an -exact contract. Complete Workshop numeric tokens outside quoted strings are -compared by their exact numeric value. A token starts with a digit and -consumes digits and `.`, so lexical spellings such as `0` and `0.0` are -equivalent while `1e3`, `.5`, and a value change such as `0.0` to `0.5` remain -regressions. Exact-output reporting still exposes presentation differences. - -For `semantic-wir`, `run_native.py` invokes the public `opy-cli compile ---format json` contract first, then invokes the feature-gated internal -`opy-compat` target with the same source project and oracle snapshot. -`opy-compat` parses only the oracle Workshop text into canonical WIR and -compares it directly with the native WIR produced by lowering through -`workshop-rs::roundtrip::equivalent`. It emits top-level harness -`compatibility.semanticWIR` evidence with both project digests. A text reparse -of the native emission is not accepted as semantic-WIR evidence, and the -public compiler API remains oracle-free. - -`run_native.py` drives the built public compiler contract for every fixture, -writes ephemeral producer results under `target/`, and delegates compiler -expectation loading, full project-input validation, stage comparison, failure -frontier comparison, and blocking classification to `diff.py`. For reference -failures, `diff.py` derives the stage and construct from the pinned oracle -diagnostics and compares that frontier with the native structured diagnostic; -different known-gap frontiers remain reported as gaps rather than matches. -Only the durable, evidence-backed -entries in `compiler-expectations.json` can produce `known-gap` or -`unsupported` in this report. -Expectation mismatches are reported as `unexpected-divergence` or `regression`. -The command exits non-zero for those mismatches and, by default, for -`inconclusive` cases; use `--allow-inconclusive` only for a local report that is -explicitly not an acceptance gate. - -Real-world fixtures retain the complete project as the integration case. A -fixture may also declare `regressions` pointing to minimized snippets under -the same directory. Each snippet records its source path, parent source, -reference status, and oracle provenance; the full project is not replaced by -the minimized case. - -The `census/workshop-feature-census` fixture is the OPY-side representation -of the future `workshop-rs#10` conformance boundary. Its feature IDs are -opaque consumer references only. Canonical Workshop identities, catalog -definitions, and validation remain owned by `workshop-rs`. - -## Commands +The Python harness drives the public CLI and, for semantic-WIR comparisons, +the feature-gated `opy-compat` comparison target: ```sh python3 -m unittest discover -s tools/overpy/tests -python3 tools/overpy/run_oracle.py --update python3 tools/overpy/run_oracle.py cargo build --locked -p opy-cli --bin opy-cli cargo build --locked -p opy-cli --features compatibility --bin opy-compat python3 tools/overpy/run_native.py \ --binary target/debug/opy-cli \ --semantic-binary target/debug/opy-compat -python3 tools/overpy/diff.py --results /path/to/results --report target/opy-rs-differential-report.json -``` - -Use `--update` only when intentionally accepting a change from the pinned -oracle. Review the resulting snapshot and fixture provenance in the same -change. A normal run fails when a snapshot is missing, the expected status is -wrong, or normalized output/diagnostics differ. - -The harness tests (`tools/overpy/tests/`) run without the oracle installed: -`test_runner.py` (normalization + corpus metadata/snapshot validity), -and `test_diff.py` (differential result contract). -The wright-side npm-packaging tests are not ported (they test wright's release -tooling). The manifest probe validator -(`crates/opy-rs/src/manifest/probes/validate.py`) is source implementation-owned and -runs standalone against the pinned oracle (Node + pnpm required), like -`run_oracle.py`; it is not part of the oracle-less harness suite. - -## Native differential runner (issue #7) - -The native source implementation side of the differential contract lives in -`crates/opy-rs/tests/differential.rs` and runs in `cargo test` with no -Node or OverPy installed: - -```sh -cargo test -p opy-rs --test differential ``` -It compiles every fixture through the native pipeline (preprocess → parse → -lower), verifies the Opy HIR v2 structure (validation, wire round-trip, -deterministic dump), and compares the outcome against the recorded -`oracle.json` evidence: status parity (resolve vs expected diagnostic), -ordered authored rule names (normalized to drop reference-synthesized -`Initialize …`/`Subroutine …` rules), and the pinned diagnostic code. Every -fixture has an explicit expected-outcome entry; behavior that leaves the -table fails the suite, while documented reference gaps (reference accepts, -native rejects a declared-rejected or not-yet-declared surface) are recorded -in the report as `referenceGap` entries without failing CI. - -Artifacts under `target/`: +Use `run_oracle.py --update` only when intentionally accepting a changed +result from the pinned reference. Snapshot changes must be reviewed with the +fixture input and concrete attribution/reproducibility data. -* `target/opy-differential/.native.json`: normalized native HIR - wire payload (span endpoints removed) per fixture; -* `target/opy-differential-report.json`: machine-readable per-fixture status - (`resolve` / `expected-diagnostic` / `divergence`), classification, native - diagnostic details, reference status, `referenceGap`, rule-name comparison, - and expectation evidence. +`run_native.py` writes producer results and reports under `target/`. The +comparison stages are compile status, diagnostics, exact/normalized output, +failure frontier, semantic WIR, diagnostic code, and explicitly declared +compiler contracts. Missing producer output or unavailable semantic-WIR data +is `inconclusive` and blocks the normal command. -Fixtures without an `oracle.json` are marked `skip` (reference comparison -degraded gracefully); the structural self-check and the expected-outcome -table still run. - -## Differential result contract - -`diff.py` compares each `oracle.json` with a producer result at -`//result.json`, or invokes a producer command once -per fixture: +`diff.py` can also compare any producer that writes the compile-result schema: ```sh python3 tools/overpy/diff.py \ @@ -193,33 +62,6 @@ python3 tools/overpy/diff.py \ --report target/opy-rs-differential-report.json ``` -`your-producer` is a placeholder: this repository ships no producer for -`diff.py`. The native source implementation comparison runs inside `cargo test` -(`crates/opy-rs/tests/differential.rs`) against the recorded oracle -snapshots directly. - -The producer must write a result with the same compile-focused schema as the -oracle and the fixture id. A compatibility runner may additionally provide a -top-level `compatibility.semanticWIR` value when a compiler contract requires -canonical-WIR evidence; this field is not part of `opy-cli`'s public compile -report. The command template is tokenized without a shell; `{fixture_id}`, -`{source}`, and `{result}` are the only substitutions. - -The report separates these stages: - -* `compile-status` and `diagnostics`; -* `exact-output`, using the unnormalized Workshop text; -* `normalized-output`, using the versioned snapshot normalization; and -* `semantic-wir`, when the isolated compatibility harness provides direct - native-WIR evidence. - -It also separates relationship outcomes: `match`, `known-gap`, `unsupported`, -`unexpected-divergence`, `regression`, and `inconclusive`. Known gaps are -reviewable evidence and are not counted as successful parity. - -An exact-output difference with a normalized-output match is reported as a -presentation difference. A normalized-output or semantic-WIR regression exits -1. Missing producer results or unavailable semantic evidence are -`inconclusive` and exit 2 by default, so a CI job cannot silently pass without -a producer. -Use `--allow-inconclusive` only for local contract checks. +The placeholder command is tokenized without a shell. The public compiler +report does not contain oracle input or semantic-WIR comparison data; those +belong to this isolated harness. diff --git a/tools/overpy/compiler-expectations.json b/tools/overpy/compiler-expectations.json deleted file mode 100644 index 468f6883..00000000 --- a/tools/overpy/compiler-expectations.json +++ /dev/null @@ -1,998 +0,0 @@ -{ - "schemaVersion": 1, - "contract": "compiler", - "description": "Reviewed compiler outcome expectations for the #38 OPY-to-Workshop evidence contract. Source/frontend expectations remain in differential-expectations.json.", - "evidencePolicy": { - "native": "The compiler result is compared with this separately reviewed expectation contract; it never creates its own expectation.", - "normalizedOutput": "Non-numeric text remains exact; complete Workshop numeric tokens outside quoted strings start with a digit, consume digits and '.', and are equivalent only when their numeric values are equal.", - "classification": "match requires normalized output, direct canonical-WIR equivalence, or a stable diagnostic contract. known-gap and unsupported require pinned oracle and fixture provenance evidence plus a concrete follow-up owner.", - "semanticWIR": "The runner compares native WIR directly with the pinned oracle WIR through workshop-rs::roundtrip::equivalent and verifies both complete project input digests. It never reparses native Workshop text as a substitute for native WIR." - }, - "cases": [ - { - "fixture": "synthetic/array-unique", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/array-unique/oracle.json", - "provenance:synthetic/array-unique/fixture.json", - "test:opy-rs::compiler::integration_tests::array_unique::array_unique_matches_the_pinned_canonical_wir" - ], - "owner": "opy-rs#322", - "note": "Array.unique() keeps the first occurrence of each value and lowers to the pinned canonical filtered-array WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/basic-rule", - "nativeStatus": "success", - "classification": "match", - "comparison": "normalized-output", - "evidence": [ - "test:opy-cli::compile_text_prints_only_final_workshop_to_stdout" - ], - "owner": "opy-rs#38", - "note": "Minimal OPY rule emits the pinned Workshop output through the public compiler and CLI." - }, - { - "fixture": "synthetic/chase-enums", - "nativeStatus": "success", - "classification": "match", - "comparison": "normalized-output", - "evidence": [ - "test:opy-rs::compiler::integration_tests::catalog_lowering::catalog_enum_members_lower_and_validate" - ], - "owner": "opy-rs#38", - "note": "Catalog enum members emit through the canonical catalog-backed compiler slice." - }, - { - "fixture": "synthetic/declarations-numbers", - "nativeStatus": "success", - "classification": "match", - "comparison": "normalized-output", - "evidence": [ - "oracle:synthetic/declarations-numbers/oracle.json" - ], - "owner": "opy-rs#38", - "note": "The declaration fixture has a stable normalized Workshop result under the bounded compiler surface." - }, - { - "fixture": "synthetic/for-range-agentlab", - "nativeStatus": "success", - "classification": "match", - "comparison": "normalized-output", - "evidence": [ - "oracle:synthetic/for-range-agentlab/oracle.json" - ], - "owner": "opy-rs#38", - "note": "The supported global range binder has a stable normalized Workshop result." - }, - { - "fixture": "synthetic/project-main-file", - "nativeStatus": "success", - "classification": "match", - "comparison": "normalized-output", - "evidence": [ - "oracle:synthetic/project-main-file/oracle.json" - ], - "owner": "opy-rs#38", - "note": "mainFile scope remains covered by the bounded compiler output contract." - }, - { - "fixture": "synthetic/project-entry", - "nativeStatus": "success", - "classification": "match", - "comparison": "normalized-output", - "evidence": [ - "oracle:synthetic/project-entry/oracle.json", - "provenance:synthetic/project-entry/fixture.json" - ], - "owner": "opy-rs#129", - "note": "An included-file mainFile directive is accepted at the file start without changing the declared root entry point." - }, - { - "fixture": "synthetic/legacy-entry-include", - "nativeStatus": "success", - "classification": "match", - "comparison": "compiler-contract", - "evidence": [ - "oracle:synthetic/legacy-entry-include/oracle.json", - "provenance:synthetic/legacy-entry-include/fixture.json" - ], - "owner": "opy-rs#258", - "note": "The minimized legacy entry topology resolves a parent-relative include from the latest included file base; compiler output parity is outside this preprocessing contract." - }, - { - "fixture": "synthetic/compiler-vertical-slice", - "nativeStatus": "success", - "classification": "match", - "comparison": "normalized-output", - "evidence": [ - "test:opy-rs::compiler::tests::vertical_slice_preserves_source_files_spans_and_emits_workshop" - ], - "owner": "opy-rs#38", - "note": "Issue #35 is the public source-to-WIR-to-Workshop vertical slice." - }, - { - "fixture": "synthetic/receiver-calls", - "nativeStatus": "success", - "classification": "match", - "comparison": "normalized-output", - "evidence": [ - "test:opy-rs::compiler::integration_tests::catalog_lowering::catalog_backed_receiver_calls_match_the_pinned_oracle" - ], - "owner": "opy-rs#38", - "note": "The exercised catalog-backed receiver call is canonical-WIR equivalent to the pinned oracle." - }, - { - "fixture": "synthetic/control-flow", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/control-flow/oracle.json", - "provenance:synthetic/control-flow/fixture.json" - ], - "owner": "opy-rs#89", - "note": "The native debug helper lowers to the pinned canonical HUD action and is directly equivalent to the oracle.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/switch-break", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/switch-break/oracle.json", - "provenance:synthetic/switch-break/fixture.json" - ], - "owner": "opy-rs#89", - "note": "The native switch-break lowering is directly equivalent to the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/compiler-structure", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "semanticEquivalent": true, - "evidence": [ - "oracle:synthetic/compiler-structure/oracle.json", - "provenance:synthetic/compiler-structure/fixture.json", - "test:opy-rs::compiler::tests::compiler_structure_matches_the_pinned_oracle" - ], - "owner": "opy-rs#38", - "note": "Issue #40 structural lowering is independently checked against the pinned oracle and WIR invariants." - }, - { - "fixture": "synthetic/primitive-lowering", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/primitive-lowering/oracle.json", - "provenance:synthetic/primitive-lowering/fixture.json" - ], - "owner": "opy-rs#89", - "note": "The native #46 primitive lowering is directly equivalent to the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/wait-optimization", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/wait-optimization/oracle.json", - "provenance:synthetic/wait-optimization/fixture.json", - "test:opy-rs::compiler::integration_tests::wait_lowering::optimized_wait_forms_match_the_pinned_oracle" - ], - "owner": "opy-rs#282", - "note": "The native wait lowering preserves pinned OverPy optimizeForSize semantics for omitted, sub-default, and one-second durations while retaining the wait behavior argument.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/control-flow-lowering", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/control-flow-lowering/oracle.json", - "provenance:synthetic/control-flow-lowering/fixture.json" - ], - "owner": "opy-rs#89", - "note": "The native #47 control-flow lowering is directly equivalent to the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/conditional-forward-gotos", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/conditional-forward-gotos/oracle.json", - "provenance:synthetic/conditional-forward-gotos/fixture.json", - "test:opy-rs::compiler::integration_tests::control_flow::conditional_forward_gotos_match_the_pinned_oracle" - ], - "owner": "opy-rs#319", - "note": "Conditional forward gotos preserve pinned OverPy branch structure, full structured-action distances, and single condition evaluation.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/control-flow-328", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/control-flow-328/oracle.json", - "provenance:synthetic/control-flow-328/fixture.json", - "test:opy-rs::compiler::integration_tests::control_flow::issue_328_control_flow_matches_the_pinned_oracle" - ], - "owner": "opy-rs#328", - "note": "Issue #328 continue, do-while, dynamic loc+, and RULE_START forms lower to the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/switch-order", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/switch-order/oracle.json", - "provenance:synthetic/switch-order/fixture.json" - ], - "owner": "opy-rs#89", - "note": "The native authored switch order is directly equivalent to the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/switch-structured-target", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/switch-structured-target/oracle.json", - "provenance:synthetic/switch-structured-target/fixture.json" - ], - "owner": "opy-rs#89", - "note": "The native structured switch targets are directly equivalent to the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/do-while-break", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "semanticEquivalent": true, - "evidence": [ - "oracle:synthetic/do-while-break/oracle.json", - "provenance:synthetic/do-while-break/fixture.json", - "test:opy-rs::compiler::integration_tests::control_flow::do_while_break_shapes_match_the_pinned_oracle" - ], - "owner": "opy-rs#38", - "note": "Supported do-while break shapes are independently checked through canonical WIR equivalence." - }, - { - "fixture": "synthetic/settings", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "semanticEquivalent": true, - "evidence": [ - "oracle:synthetic/settings/oracle.json", - "provenance:synthetic/settings/fixture.json", - "test:opy-rs::compiler::tests::settings_lower_through_workshop_owned_emission" - ], - "owner": "opy-rs#38", - "note": "Settings emission is covered by the compiler's Workshop-owned structural comparison." - }, - { - "fixture": "synthetic/do-while-invalid", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "test:opy-rs::compiler::integration_tests::control_flow::invalid_do_while_placement_is_source_attributed" - ], - "owner": "opy-rs#38", - "note": "Invalid do-while placement is a stable source-attributed compiler failure.", - "failureClass": "frontend", - "diagnosticCode": "do-while-placement" - }, - { - "fixture": "synthetic/dictionary-lookup", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/dictionary-lookup/oracle.json", - "provenance:synthetic/dictionary-lookup/fixture.json", - "test:opy-rs::compiler::integration_tests::dictionary_lowering::literal_dictionary_lookup_matches_the_pinned_oracle" - ], - "owner": "opy-rs#145", - "note": "Literal dictionary lookups with literal keys are folded to their selected value, or to Null on a literal miss, matching the pinned oracle; dictionary-indexed assignment targets remain outside this bounded lowering path.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/postfix-assignment", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/postfix-assignment/oracle.json", - "provenance:synthetic/postfix-assignment/fixture.json", - "test:opy-rs::compiler::integration_tests::postfix_assignments::postfix_assignments_match_the_pinned_oracle" - ], - "owner": "opy-rs#59", - "note": "Postfix ++/-- assignments lower to canonical Add/Subtract modifications for global, player, and single-level indexed variables.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/postfix-prefix-invalid", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/postfix-prefix-invalid/oracle.json", - "provenance:synthetic/postfix-prefix-invalid/fixture.json", - "test:opy-rs::compiler::integration_tests::postfix_assignments::rejected_prefix_increment_form_has_stable_source_diagnostic" - ], - "owner": "opy-rs#59", - "note": "The pinned oracle rejects prefix ++ and native reports the stable source-attributed frontend diagnostic; prefix --x remains valid consecutive unary-minus syntax.", - "failureClass": "frontend", - "diagnosticCode": "parse-error" - }, - { - "fixture": "synthetic/postfix-embedded-invalid", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/postfix-embedded-invalid/oracle.json", - "provenance:synthetic/postfix-embedded-invalid/fixture.json", - "test:opy-rs::compiler::integration_tests::postfix_assignments::rejected_embedded_postfix_form_has_stable_source_diagnostic" - ], - "owner": "opy-rs#59", - "note": "The pinned oracle independently rejects the embedded postfix form and native reports the stable source-attributed frontend diagnostic.", - "failureClass": "frontend", - "diagnosticCode": "parse-error" - }, - { - "fixture": "synthetic/range-player-variable", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/range-player-variable/oracle.json", - "provenance:synthetic/range-player-variable/fixture.json", - "test:opy-rs::compiler::integration_tests::range_bindings::player_variable_range_binder_matches_the_pinned_oracle" - ], - "owner": "opy-rs#65", - "note": "The pinned hostPlayer.I range binder lowers to the existing canonical For Player Variable WIR contract and is directly equivalent to the oracle.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/range-invalid-binder", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/range-invalid-binder/oracle.json", - "provenance:synthetic/range-invalid-binder/fixture.json", - "test:opy-rs::compiler::integration_tests::semantic_diagnostics::semantic_boundary_errors_are_reported_before_lowering" - ], - "owner": "opy-rs#162", - "note": "The pinned oracle and native source semantic boundary reject a non-variable range binder with a source-attributed diagnostic before canonical lowering.", - "failureClass": "frontend", - "diagnosticCode": "invalid-range-binder" - }, - { - "fixture": "synthetic/member-values", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/member-values/oracle.json", - "provenance:synthetic/member-values/fixture.json", - "test:opy-rs::compiler::integration_tests::member_values::is_dummy_member_lowers_to_the_catalog_value_in_canonical_wir" - ], - "owner": "opy-rs#113", - "note": "The catalog-backed eventPlayer.isDummy() predicate lowers to a canonical WIR value call equivalent to the pinned oracle.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/hud-subheader", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/hud-subheader/oracle.json", - "provenance:synthetic/hud-subheader/fixture.json", - "test:opy-rs::compiler::integration_tests::hud::hud_subheader_matches_the_pinned_canonical_wir" - ], - "owner": "opy-rs#114", - "note": "The shared hudSubheader action lowers to canonical Create HUD Text WIR with the pinned argument order and visibility semantics.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/member-angle", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/member-angle/oracle.json", - "provenance:synthetic/member-angle/fixture.json", - "test:opy-rs::compiler::integration_tests::member_angles::horizontal_facing_angle_member_matches_the_pinned_canonical_wir" - ], - "owner": "opy-rs#130", - "note": "The catalog-backed eventPlayer.getHorizontalFacingAngle() member value lowers to canonical WIR equivalent to the pinned oracle.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/hud-visibility", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/hud-visibility/oracle.json", - "provenance:synthetic/hud-visibility/fixture.json", - "test:opy-rs::compiler::integration_tests::hud_visibility::never_maps_to_visible_never_in_canonical_wir" - ], - "owner": "opy-rs#131", - "note": "The OPY SpecVisibility.NEVER spelling lowers to the canonical VISIBLE_NEVER enum member and is equivalent to the pinned oracle WIR; unrelated SpecVisibility members remain unchanged.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/switch-break-unsupported", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/switch-break-unsupported/oracle.json", - "provenance:synthetic/switch-break-unsupported/fixture.json", - "test:opy-rs::compiler::integration_tests::control_flow::nested_switch_break_matches_upstream_noop_elision" - ], - "owner": "opy-rs#47", - "note": "Nested switch breaks are represented by the upstream Else marker; a top-level switch with no observable actions is elided like the pinned oracle.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/switch-multiple-break", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "semanticEquivalent": true, - "evidence": [ - "oracle:synthetic/switch-multiple-break/oracle.json", - "oracle:synthetic/switch-multiple-break/semantic-oracle.json", - "provenance:synthetic/switch-multiple-break/fixture.json", - "test:opy-rs::compiler::integration_tests::control_flow::multiple_switch_breaks_match_independent_semantic_oracle", - "test:opy-rs::compiler::integration_tests::control_flow::pinned_overpy_switch_action_trace" - ], - "owner": "opy-rs#172", - "note": "The pinned OverPy snapshot is preserved as source evidence; its successive Else switch encoding is normalized into the canonical Workshop semantic oracle used by the compatibility gate. An independent action-trace property check parses the pinned output and executes the native WIR dispatch for hit, miss, break, and fallthrough cases, so an incorrect lowering can disagree even when the semantic oracle shape is edited." - }, - { - "fixture": "census/workshop-feature-census", - "nativeStatus": "success", - "classification": "match", - "comparison": "normalized-output", - "evidence": [ - "oracle:census/workshop-feature-census/oracle.json", - "provenance:census/workshop-feature-census/fixture.json", - "contract:lowering.canonical-wir/vertical-slice" - ], - "owner": "opy-rs#38", - "note": "The bounded feature census matches the pinned oracle through the normalized compiler output contract." - }, - { - "fixture": "real-world/6v6-adjustments", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:real-world/6v6-adjustments/oracle.json", - "provenance:real-world/6v6-adjustments/fixture.json" - ], - "owner": "opy-rs#167", - "note": "The source frontend reaches the pinned semantic unknown-member failure frontier, matching the reference oracle.", - "failureClass": "frontend", - "diagnosticCode": "unknown-member" - }, - { - "fixture": "real-world/overpy-broken-weapons", - "nativeStatus": "success", - "classification": "known-gap", - "comparison": "semantic-wir", - "evidence": [ - "oracle:real-world/overpy-broken-weapons/oracle.json", - "provenance:real-world/overpy-broken-weapons/fixture.json" - ], - "owner": "opy-rs#157", - "note": "The full real-world project now compiles through the final #141 grammar and #144 builtin/member/enum residuals; canonical WIR equivalence remains an explicit compiler boundary for the existing real-world lowering differences.", - "semanticEquivalent": false - }, - { - "fixture": "real-world/overpy-cake", - "nativeStatus": "success", - "classification": "known-gap", - "comparison": "semantic-wir", - "evidence": [ - "oracle:real-world/overpy-cake/oracle.json", - "provenance:real-world/overpy-cake/fixture.json" - ], - "owner": "workshop-rs#124", - "note": "The workshop-rs 0.3 catalog resolves `Visible To` through the canonical call-signature domain, so the native WIR can be compared directly with the pinned oracle; remaining arithmetic and implicit-value lowering differences keep this real-world case an explicit compiler gap.", - "semanticEquivalent": false - }, - { - "fixture": "real-world/overpy-client-to-server", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:real-world/overpy-client-to-server/oracle.json", - "provenance:real-world/overpy-client-to-server/fixture.json" - ], - "owner": "opy-rs#157", - "note": "The client-to-server project lowers its print text carrier and negated predicate conditions to the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "real-world/overpy-cronch", - "nativeStatus": "success", - "classification": "known-gap", - "comparison": "semantic-wir", - "evidence": [ - "oracle:real-world/overpy-cronch/oracle.json", - "provenance:real-world/overpy-cronch/fixture.json" - ], - "owner": "opy-rs#157", - "note": "The full real-world project now compiles through the final #141 grammar and #144 builtin/member/enum residuals; canonical WIR equivalence remains an explicit compiler boundary for the existing real-world lowering differences.", - "semanticEquivalent": false - }, - { - "fixture": "real-world/overpy-crosshair", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:real-world/overpy-crosshair/oracle.json", - "provenance:real-world/overpy-crosshair/fixture.json" - ], - "owner": "opy-rs#157", - "note": "The crosshair project lowers its negated button conditions and HUD calls to the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "real-world/overpy-inputhud", - "nativeStatus": "success", - "classification": "known-gap", - "comparison": "semantic-wir", - "evidence": [ - "oracle:real-world/overpy-inputhud/oracle.json", - "provenance:real-world/overpy-inputhud/fixture.json" - ], - "owner": "opy-rs#157", - "note": "The full real-world project now compiles through the final #141 grammar and #144 builtin/member/enum residuals; canonical WIR equivalence remains an explicit compiler boundary for the existing real-world lowering differences.", - "semanticEquivalent": false - }, - { - "fixture": "real-world/overpy-meipocalypse", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:real-world/overpy-meipocalypse/oracle.json", - "provenance:real-world/overpy-meipocalypse/fixture.json" - ], - "owner": "opy-rs#142", - "note": "Preprocessing reaches the pinned missing-script failure frontier for generateWalls.js, matching the reference oracle.", - "failureClass": "frontend", - "diagnosticCode": "script-not-found" - }, - { - "fixture": "real-world/overpy-parabola", - "nativeStatus": "success", - "classification": "known-gap", - "comparison": "semantic-wir", - "evidence": [ - "oracle:real-world/overpy-parabola/oracle.json", - "provenance:real-world/overpy-parabola/fixture.json" - ], - "owner": "opy-rs#157", - "note": "The full real-world project now compiles through the final #141 grammar and #144 builtin/member/enum residuals; canonical WIR equivalence remains an explicit compiler boundary for the existing real-world lowering differences.", - "semanticEquivalent": false - }, - { - "fixture": "real-world/overpy-pixelart", - "nativeStatus": "success", - "classification": "known-gap", - "comparison": "semantic-wir", - "evidence": [ - "oracle:real-world/overpy-pixelart/oracle.json", - "provenance:real-world/overpy-pixelart/fixture.json" - ], - "owner": "opy-rs#157", - "note": "The source compiles, but full real-world Workshop parity is outside the declared #38 compiler contract.", - "semanticEquivalent": false - }, - { - "fixture": "real-world/overpy-santa", - "nativeStatus": "success", - "classification": "known-gap", - "comparison": "semantic-wir", - "evidence": [ - "oracle:real-world/overpy-santa/oracle.json", - "provenance:real-world/overpy-santa/fixture.json" - ], - "owner": "opy-rs#157", - "note": "The full real-world project now compiles through the final #141 grammar and #144 builtin/member/enum residuals; canonical WIR equivalence remains an explicit compiler boundary for the existing real-world lowering differences.", - "semanticEquivalent": false - }, - { - "fixture": "real-world/overpy-zencopter", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:real-world/overpy-zencopter/oracle.json", - "provenance:real-world/overpy-zencopter/fixture.json" - ], - "owner": "opy-rs#141", - "note": "The native lexer stops at the invalid content before string frontier on arena, matching the reference oracle.", - "failureClass": "frontend", - "diagnosticCode": "lex-error" - }, - { - "fixture": "real-world/ow1-emulator", - "nativeStatus": "failure", - "classification": "known-gap", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:real-world/ow1-emulator/oracle.json", - "provenance:real-world/ow1-emulator/fixture.json" - ], - "owner": "opy-rs#47", - "note": "An independent compiler gate rerun after the catalog-link and parser-boundary fixes advances the project past the prior parse-error frontier; it now reaches the existing canonical WIR limitation for multiple switch breaks with later reachable actions.", - "failureClass": "integration", - "diagnosticCode": "unsupported-integration-surface" - }, - { - "fixture": "synthetic/chase-condition-agentlab", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/chase-condition-agentlab/oracle.json", - "provenance:synthetic/chase-condition-agentlab/fixture.json" - ], - "owner": "workshop-rs#124", - "note": "The workshop-rs 0.3 catalog resolves `None` through the canonical call-signature domain, so the native WIR can be compared directly with the pinned oracle.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/chase-keywords", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/chase-keywords/oracle.json", - "provenance:synthetic/chase-keywords/fixture.json", - "test:opy-rs::compiler::integration_tests::catalog_lowering::chase_keyword_forms_match_the_pinned_oracle" - ], - "owner": "opy-rs#157", - "note": "Keyword binding, literal count folding, text carriers, and contextual chase lowering match the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/declarations-rules", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/declarations-rules/oracle.json", - "provenance:synthetic/declarations-rules/fixture.json" - ], - "owner": "opy-rs#94", - "note": "Declaration, subroutine, receiver, and print lowering match the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/expressions-values", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/expressions-values/oracle.json", - "provenance:synthetic/expressions-values/fixture.json" - ], - "owner": "opy-rs#94", - "note": "Expression lowering, literal arithmetic folding, array values, and debug output match the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/diagnostics", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/diagnostics/oracle.json", - "provenance:synthetic/diagnostics/fixture.json" - ], - "owner": "opy-rs#94", - "note": "The compiler preserves the pinned frontend parse-error code and source-attributed diagnostic contract.", - "failureClass": "frontend", - "diagnosticCode": "parse-error" - }, - { - "fixture": "synthetic/syntax-invalid", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/syntax-invalid/oracle.json", - "provenance:synthetic/syntax-invalid/fixture.json", - "test:opy-rs::compiler::integration_tests::frontend_diagnostics::frontend_diagnostics_are_public_and_source_attributed" - ], - "owner": "opy-rs#93", - "note": "The public compiler reports the pinned malformed syntax as a source-attributed parse diagnostic.", - "failureClass": "frontend", - "diagnosticCode": "parse-error" - }, - { - "fixture": "synthetic/string-modifiers", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/string-modifiers/oracle.json", - "provenance:synthetic/string-modifiers/fixture.json", - "test:opy-rs::compiler::integration_tests::string_modifiers::string_modifier_lowering_matches_the_pinned_oracle" - ], - "owner": "opy-rs#145", - "note": "The compiler lowers the evidenced f/w/b/c literal string modifiers to canonical Custom String values; l/t remain translation-carried source constructs.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/syntax-surface", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/syntax-surface/oracle.json", - "provenance:synthetic/syntax-surface/fixture.json", - "test:opy-rs::compiler::integration_tests::syntax_lowering::broad_supported_syntax_lowering_reaches_canonical_wir" - ], - "owner": "opy-rs#157", - "note": "The supported pure-OPY syntax probe lowers through canonical WIR, including switch jump tables, do-while loops, string modifiers, dictionary lookups, and literal membership folding, matching the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/directives", - "nativeStatus": "failure", - "classification": "unsupported", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/directives/oracle.json", - "provenance:synthetic/directives/fixture.json" - ], - "owner": "opy-rs#271", - "note": "Unsupported backend replacement directives are explicitly rejected with a source-attributed diagnostic under #271.", - "failureClass": "integration", - "diagnosticCode": "backend-directive-unsupported" - }, - { - "fixture": "synthetic/duplicate-rule-diagnostic", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/duplicate-rule-diagnostic/oracle.json", - "provenance:synthetic/duplicate-rule-diagnostic/fixture.json", - "test:opy-rs::compiler::integration_tests::diagnostic_frontiers::duplicate_rule_name_reaches_the_source_frontier" - ], - "owner": "opy-rs#167", - "note": "An ordinary rule @Name annotation reaches the pinned duplicate-rule-name semantic frontier with source attribution.", - "failureClass": "frontend", - "diagnosticCode": "duplicate-rule-name" - }, - { - "fixture": "synthetic/translations-invalid", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/translations-invalid/oracle.json", - "provenance:synthetic/translations-invalid/fixture.json", - "test:opy-rs::compiler::integration_tests::frontend_diagnostics::frontend_diagnostics_are_public_and_source_attributed" - ], - "owner": "opy-rs#93", - "note": "The public compiler reports the invalid translation code as a source-attributed frontend diagnostic.", - "failureClass": "frontend", - "diagnosticCode": "translations-invalid" - }, - { - "fixture": "synthetic/include-scope", - "nativeStatus": "success", - "classification": "match", - "comparison": "compiler-contract", - "evidence": [ - "oracle:synthetic/include-scope/oracle.json", - "provenance:synthetic/include-scope/fixture.json" - ], - "owner": "opy-rs#288", - "note": "Strict optimization is accepted in the nested include state probe; its pass-only output remains outside the compiler parity contract." - }, - { - "fixture": "synthetic/optimize-strict", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/optimize-strict/oracle.json", - "provenance:synthetic/optimize-strict/fixture.json", - "test:opy-rs::compiler::integration_tests::directive_lowering::strict_optimizer_fixture_matches_the_pinned_canonical_wir" - ], - "owner": "opy-rs#288", - "note": "The pinned strict optimizer probe retains Add, Multiply, Divide, Or, and And expressions that non-strict OverPy optimization would rewrite.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/directives-scoped", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/directives-scoped/oracle.json", - "provenance:synthetic/directives-scoped/fixture.json", - "test:opy-rs::compiler::integration_tests::directive_lowering::translated_implicit_subroutine_fixture_matches_the_pinned_oracle" - ], - "owner": "opy-rs#145", - "note": "The compiler materializes the implicit def subroutine declaration and the translation helper initialization required by the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/indexed-assignment-nested", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/indexed-assignment-nested/oracle.json", - "provenance:synthetic/indexed-assignment-nested/fixture.json", - "test:opy-rs::compiler::integration_tests::indexed_assignments::nested_indexed_assignments_match_the_pinned_oracle" - ], - "owner": "opy-rs#145", - "note": "Nested indexed assignments, including compound modification and three-level replacement, lower through canonical array slices and match the pinned canonical WIR.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/collection-mutation-328", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/collection-mutation-328/oracle.json", - "provenance:synthetic/collection-mutation-328/fixture.json", - "test:opy-rs::compiler::integration_tests::indexed_assignments::collection_deletion_matches_the_pinned_oracle" - ], - "owner": "opy-rs#328", - "note": "Issue #328 collection deletion covers one- through four-index global targets through canonical indexed mutation lowering.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/collection-mutation-328-invalid", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/collection-mutation-328-invalid/oracle.json", - "provenance:synthetic/collection-mutation-328-invalid/fixture.json", - "test:opy-rs::compiler::integration_tests::semantic_diagnostics::semantic_boundary_errors_are_reported_before_lowering" - ], - "owner": "opy-rs#328", - "note": "OverPy 9.7.10 accepts four source indexes but rejects the fifth; native preserves the five-index frontend boundary.", - "failureClass": "frontend", - "diagnosticCode": "four-dimensional-delete" - }, - { - "fixture": "synthetic/collection-mutation-328-random-invalid", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/collection-mutation-328-random-invalid/oracle.json", - "provenance:synthetic/collection-mutation-328-random-invalid/fixture.json", - "test:opy-rs::compiler::integration_tests::indexed_assignments::nested_random_delete_reaches_the_pinned_boundary" - ], - "owner": "opy-rs#328", - "note": "OverPy 9.7.10 rejects a random outer index in a three-index delete; native preserves the same lowering boundary.", - "failureClass": "integration", - "diagnosticCode": "unsupported-integration-surface" - }, - { - "fixture": "synthetic/collection-mutation-328-random-player-invalid", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/collection-mutation-328-random-player-invalid/oracle.json", - "provenance:synthetic/collection-mutation-328-random-player-invalid/fixture.json", - "test:opy-rs::compiler::integration_tests::indexed_assignments::nested_random_player_receiver_delete_reaches_the_pinned_boundary" - ], - "owner": "opy-rs#328", - "note": "OverPy 9.7.10 rejects a random player-variable receiver in a three-index delete; native preserves the same lowering boundary.", - "failureClass": "integration", - "diagnosticCode": "unsupported-integration-surface" - }, - { - "fixture": "synthetic/indexed-assignment-4d-invalid", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/indexed-assignment-4d-invalid/oracle.json", - "provenance:synthetic/indexed-assignment-4d-invalid/fixture.json", - "test:opy-rs::compiler::integration_tests::semantic_diagnostics::semantic_boundary_errors_are_reported_before_lowering" - ], - "owner": "opy-rs#162", - "note": "The pinned OverPy implementation rejects four-dimensional indexed assignments; native source semantics now preserves that boundary before canonical lowering.", - "failureClass": "frontend", - "diagnosticCode": "four-dimensional-assignment" - }, - { - "fixture": "synthetic/strings-and-lambda", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/strings-and-lambda/oracle.json", - "provenance:synthetic/strings-and-lambda/fixture.json" - ], - "owner": "opy-rs#103", - "note": "The demonstrated sorted-array key lambda lowers to canonical Sorted Array WIR and matches the pinned oracle.", - "semanticEquivalent": true - }, - { - "fixture": "synthetic/lambda-invalid", - "nativeStatus": "failure", - "classification": "match", - "comparison": "diagnostic-code", - "evidence": [ - "oracle:synthetic/lambda-invalid/oracle.json", - "provenance:synthetic/lambda-invalid/fixture.json", - "test:opy-rs::compiler::integration_tests::frontend_diagnostics::frontend_diagnostics_are_public_and_source_attributed" - ], - "owner": "opy-rs#93", - "note": "The public compiler reports standalone lambda use as a source-attributed lambda-context diagnostic.", - "failureClass": "frontend", - "diagnosticCode": "lambda-context" - }, - { - "fixture": "synthetic/preprocessing", - "nativeStatus": "success", - "classification": "known-gap", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/preprocessing/oracle.json", - "provenance:synthetic/preprocessing/fixture.json" - ], - "owner": "opy-rs#85", - "note": "Included macro expansion now reaches canonical WIR through #85, but the pinned oracle's macro-emission semantics remain a separately tracked compiler gap.", - "semanticEquivalent": false - }, - { - "fixture": "synthetic/receiver-playervar", - "nativeStatus": "success", - "classification": "match", - "comparison": "semantic-wir", - "evidence": [ - "oracle:synthetic/receiver-playervar/oracle.json", - "provenance:synthetic/receiver-playervar/fixture.json" - ], - "owner": "opy-rs#94", - "note": "Bare default-variable receivers preserve global identity and player-variable member lowering matches the pinned canonical WIR.", - "semanticEquivalent": true - } - ] -} diff --git a/tools/overpy/diff.py b/tools/overpy/diff.py index 0f342344..80be0759 100644 --- a/tools/overpy/diff.py +++ b/tools/overpy/diff.py @@ -21,8 +21,6 @@ ROOT = Path(__file__).resolve().parents[2] DEFAULT_FIXTURES = ROOT / "crates/opy-rs/tests/fixtures/corpus" DEFAULT_REPORT = ROOT / "target" / "opy-rs-differential-report.json" -DEFAULT_EXPECTATIONS = ROOT / "crates/opy-rs/tests/differential-expectations.json" -DEFAULT_COMPILER_EXPECTATIONS = ROOT / "tools/overpy/compiler-expectations.json" EXPECTED_NATIVE_STATUSES = {"success", "failure"} EXPECTED_CLASSIFICATIONS = {"match", "known-gap", "unsupported"} @@ -32,10 +30,9 @@ "diagnostic-code", "compiler-contract", } -CONCRETE_GAP_OWNER = re.compile(r"(?:opy-rs|workshop-rs)#[1-9][0-9]*") NUMBER_TOKEN = re.compile(r"\d[\d.]*") -# These are evidence rules for the pinned oracle's recorded diagnostics, not a +# These rules classify the pinned oracle's recorded diagnostics, not a # support inventory. The native side uses structured diagnostic codes below; # the reference side intentionally derives its frontier from oracle text. REFERENCE_FRONTIER_RULES = ( @@ -122,153 +119,71 @@ def fixture_ids(fixtures_root: Path) -> list[str]: return ids -def load_expectations(path: Path = DEFAULT_EXPECTATIONS) -> dict[str, dict[str, Any]]: - data = load_json(path) - if data.get("schemaVersion") != 1: - raise DiffError(f"unsupported differential expectation schema: {path}") - cases = data.get("cases") - if not isinstance(cases, list) or not cases: - raise DiffError(f"differential expectations must contain cases: {path}") - +def _load_test_expectations( + fixtures_root: Path, + test_name: str, +) -> dict[str, dict[str, Any]]: by_fixture: dict[str, dict[str, Any]] = {} - for case in cases: - if not isinstance(case, dict): - raise DiffError(f"differential expectation must be an object: {path}") - fixture = case.get("fixture") + for metadata_path in sorted(fixtures_root.glob("**/fixture.json")): + metadata = load_json(metadata_path) + fixture = metadata.get("id") if not isinstance(fixture, str) or not fixture: - raise DiffError(f"differential expectation fixture is invalid: {path}") - if fixture in by_fixture: - raise DiffError(f"duplicate differential expectation: {fixture}") - native_status = case.get("nativeStatus") - if native_status not in EXPECTED_NATIVE_STATUSES: - raise DiffError(f"{fixture}: nativeStatus must be success or failure") - classification = case.get("classification") - if classification not in EXPECTED_CLASSIFICATIONS: - raise DiffError( - f"{fixture}: classification must be match, known-gap, or unsupported" - ) - evidence = case.get("evidence") - if not isinstance(evidence, list) or not evidence or not all( - isinstance(item, str) and item for item in evidence - ): - raise DiffError(f"{fixture}: evidence must be a non-empty string array") - note = case.get("note") - if not isinstance(note, str) or not note: - raise DiffError(f"{fixture}: note must be a non-empty string") - if not isinstance(case.get("ruleNames"), bool): - raise DiffError(f"{fixture}: ruleNames must be boolean") + raise DiffError(f"fixture id is invalid: {metadata_path}") + tests = metadata.get("tests") + expectation = tests.get(test_name) if isinstance(tests, dict) else None + if not isinstance(expectation, dict): + raise DiffError(f"{fixture}: tests.{test_name} must be an object") + case = dict(expectation) + case["fixture"] = fixture by_fixture[fixture] = case + if not by_fixture: + raise DiffError(f"no {test_name} expectations found under {fixtures_root}") return by_fixture -def load_compiler_expectations( - path: Path = DEFAULT_COMPILER_EXPECTATIONS, +def load_expectations( + fixtures_root: Path = DEFAULT_FIXTURES, ) -> dict[str, dict[str, Any]]: - data = load_json(path) - if data.get("schemaVersion") != 1 or data.get("contract") != "compiler": - raise DiffError(f"unsupported compiler expectation schema: {path}") - cases = data.get("cases") - if not isinstance(cases, list) or not cases: - raise DiffError(f"compiler expectations must contain cases: {path}") + expectations = _load_test_expectations(fixtures_root, "source") + for fixture, case in expectations.items(): + if case.get("nativeStatus") not in EXPECTED_NATIVE_STATUSES: + raise DiffError(f"{fixture}: source nativeStatus must be success or failure") + if case.get("relationship") not in EXPECTED_CLASSIFICATIONS: + raise DiffError(f"{fixture}: source relationship is invalid") + if not isinstance(case.get("ruleNames"), bool): + raise DiffError(f"{fixture}: source ruleNames must be boolean") + return expectations - by_fixture: dict[str, dict[str, Any]] = {} - for case in cases: - if not isinstance(case, dict): - raise DiffError(f"compiler expectation must be an object: {path}") - fixture = case.get("fixture") - if not isinstance(fixture, str) or not fixture: - raise DiffError(f"compiler expectation fixture is invalid: {path}") - if fixture in by_fixture: - raise DiffError(f"duplicate compiler expectation: {fixture}") + +def load_compiler_expectations( + fixtures_root: Path = DEFAULT_FIXTURES, +) -> dict[str, dict[str, Any]]: + expectations = _load_test_expectations(fixtures_root, "compiler") + for fixture, case in expectations.items(): native_status = case.get("nativeStatus") if native_status not in EXPECTED_NATIVE_STATUSES: - raise DiffError(f"{fixture}: nativeStatus must be success or failure") - classification = case.get("classification") - if classification not in EXPECTED_CLASSIFICATIONS: - raise DiffError( - f"{fixture}: classification must be match, known-gap, or unsupported" - ) + raise DiffError(f"{fixture}: compiler nativeStatus must be success or failure") + if case.get("relationship") not in EXPECTED_CLASSIFICATIONS: + raise DiffError(f"{fixture}: compiler relationship is invalid") comparison = case.get("comparison") if comparison not in EXPECTED_COMPILER_COMPARISONS: - raise DiffError( - f"{fixture}: comparison must be normalized-output, semantic-wir, " - "diagnostic-code, or compiler-contract" - ) - evidence = case.get("evidence") - if not isinstance(evidence, list) or not evidence or not all( - isinstance(item, str) and item for item in evidence - ): - raise DiffError(f"{fixture}: evidence must be a non-empty string array") - owner = case.get("owner") - if not isinstance(owner, str) or not CONCRETE_GAP_OWNER.fullmatch(owner): - raise DiffError( - f"{fixture}: owner must be a concrete opy-rs or workshop-rs issue" - ) - note = case.get("note") - if not isinstance(note, str) or not note: - raise DiffError(f"{fixture}: note must be a non-empty string") + raise DiffError(f"{fixture}: compiler comparison is invalid") if comparison in {"normalized-output", "semantic-wir"} and native_status != "success": raise DiffError(f"{fixture}: output comparisons require nativeStatus success") if comparison == "semantic-wir": semantic_equivalent = case.get("semanticEquivalent") if not isinstance(semantic_equivalent, bool): - raise DiffError( - f"{fixture}: semantic-wir requires boolean semanticEquivalent" - ) - required_evidence = { - f"oracle:{fixture}/oracle.json", - f"provenance:{fixture}/fixture.json", - } - missing_evidence = sorted(required_evidence - set(evidence)) - if missing_evidence: - raise DiffError( - f"{fixture}: semantic-wir expectations require concrete evidence: " - + ", ".join(missing_evidence) - ) - if classification == "match" and not semantic_equivalent: - raise DiffError( - f"{fixture}: a semantic-wir match requires semanticEquivalent=true" - ) - if classification == "known-gap" and semantic_equivalent: - raise DiffError( - f"{fixture}: a semantic-wir known-gap requires semanticEquivalent=false" - ) - if classification != "match": - required_evidence = { - f"oracle:{fixture}/oracle.json", - f"provenance:{fixture}/fixture.json", - } - missing_evidence = sorted(required_evidence - set(evidence)) - if missing_evidence: - raise DiffError( - f"{fixture}: non-match expectations require concrete evidence: " - + ", ".join(missing_evidence) - ) - fixtures_root = DEFAULT_FIXTURES - oracle_path = fixtures_root / fixture / "oracle.json" - provenance_path = fixtures_root / fixture / "fixture.json" - oracle = load_json(oracle_path) - provenance = load_json(provenance_path) - if oracle.get("fixture") != fixture: - raise DiffError(f"{fixture}: oracle evidence fixture does not match") - if provenance.get("id") != fixture: - raise DiffError(f"{fixture}: provenance evidence fixture does not match") - try: - expected_input = input_identity.project_input( - fixtures_root / fixture, - provenance, - ) - except input_identity.InputIdentityError as error: - raise DiffError(f"{fixture}: invalid provenance source graph: {error}") from error - if oracle.get("input") != expected_input: - raise DiffError(f"{fixture}: oracle evidence input graph is stale") + raise DiffError(f"{fixture}: semantic-wir requires semanticEquivalent") + if case["relationship"] == "match" and not semantic_equivalent: + raise DiffError(f"{fixture}: semantic-wir match requires equivalence") + if case["relationship"] == "known-gap" and semantic_equivalent: + raise DiffError(f"{fixture}: semantic-wir known-gap requires non-equivalence") if comparison == "diagnostic-code": if not isinstance(case.get("diagnosticCode"), str) or not case["diagnosticCode"]: raise DiffError(f"{fixture}: diagnostic-code requires diagnosticCode") if not isinstance(case.get("failureClass"), str) or not case["failureClass"]: raise DiffError(f"{fixture}: diagnostic-code requires failureClass") - by_fixture[fixture] = case - return by_fixture + return expectations def require_result_shape(result: dict[str, Any], label: str) -> None: @@ -289,7 +204,7 @@ def require_result_shape(result: dict[str, Any], label: str) -> None: if not isinstance(compile_result["workshop"], str): raise DiffError(f"{label}: compile.workshop must be a string") if "semanticWIR" in compile_result: - raise DiffError(f"{label}: compatibility evidence must not be part of compile result") + raise DiffError(f"{label}: semantic-WIR comparison must not be part of compile result") def _diagnostic_texts(compile_result: dict[str, Any]) -> list[str]: @@ -532,7 +447,7 @@ def compare_fixture( metadata_path = fixtures_root / fixture_id / "fixture.json" oracle_path = fixtures_root / fixture_id / "oracle.json" metadata = load_json(metadata_path) - expectations = load_expectations(DEFAULT_EXPECTATIONS) + expectations = load_expectations(fixtures_root) expectation = expectations.get(fixture_id) if expectation is None: raise DiffError(f"missing differential expectation: {fixture_id}") @@ -588,12 +503,12 @@ def compare_fixture( declared_reference_gap = oracle_status != expectation["nativeStatus"] if native_status_mismatch: status = "unexpected-divergence" - elif expectation["classification"] in {"known-gap", "unsupported"}: + elif expectation["relationship"] in {"known-gap", "unsupported"}: if not declared_reference_gap: raise DiffError( - f"{fixture_id}: {expectation['classification']} must differ from oracle status" + f"{fixture_id}: {expectation['relationship']} must differ from oracle status" ) - status = expectation["classification"] + status = expectation["relationship"] elif oracle_status != native_status: status = "unexpected-divergence" elif regression_stages: @@ -606,13 +521,11 @@ def compare_fixture( "fixture": fixture_id, "category": metadata.get("category", "unknown"), "status": status, - "expectedClassification": expectation["classification"], + "expectedRelationship": expectation["relationship"], "expectedNativeStatus": expectation["nativeStatus"], "referenceStatus": oracle_status, "referenceGap": reference_gap, "declaredReferenceGap": declared_reference_gap, - "evidence": expectation["evidence"], - "note": expectation["note"], "regressionStages": regression_stages, "differenceStages": differences, "inconclusiveStages": inconclusive, @@ -643,10 +556,8 @@ def compare_compiler_fixture( "fixture": fixture_id, "category": metadata.get("category", "unknown"), "status": "inconclusive", - "expectedClassification": expectation["classification"], + "expectedRelationship": expectation["relationship"], "expectedNativeStatus": expectation["nativeStatus"], - "evidence": expectation["evidence"], - "owner": expectation["owner"], "reason": f"missing producer result: {path}", "stages": [], } @@ -720,7 +631,7 @@ def compare_compiler_fixture( stage( "semantic-wir", "inconclusive", - reason="producer did not emit executable canonical-WIR evidence", + reason="producer did not emit an executable canonical-WIR comparison", ) ) status = "inconclusive" @@ -737,7 +648,7 @@ def compare_compiler_fixture( equivalent = semantic.get("equivalent") is True reference_error = semantic.get("referenceError") reference_parsed = reference_error is None - evidence_matches = ( + comparison_matches = ( reference_parsed and input_matches and reference_matches @@ -747,9 +658,9 @@ def compare_compiler_fixture( expected_equivalent = expectation["semanticEquivalent"] if not reference_parsed: semantic_status = "inconclusive" - elif evidence_matches and expected_equivalent: + elif comparison_matches and expected_equivalent: semantic_status = "match" - elif evidence_matches: + elif comparison_matches: semantic_status = "accepted-gap" else: semantic_status = "regression" @@ -769,9 +680,9 @@ def compare_compiler_fixture( "inconclusive" if not reference_parsed else "match" - if evidence_matches and expectation["classification"] == "match" - else expectation["classification"] - if evidence_matches + if comparison_matches and expectation["relationship"] == "match" + else expectation["relationship"] + if comparison_matches else "regression" ) elif contract == "diagnostic-code": @@ -799,13 +710,12 @@ def compare_compiler_fixture( "compiler-contract", "accepted-gap", reason="compiler parity is outside the declared compiler contract", - evidence=expectation["evidence"], ) ) - status = expectation["classification"] + status = expectation["relationship"] - if expectation["classification"] != "match" and status == "match": - status = expectation["classification"] + if expectation["relationship"] != "match" and status == "match": + status = expectation["relationship"] frontier_stage = next( (item for item in stages if item["name"] == "failure-frontier"), None, @@ -815,8 +725,8 @@ def compare_compiler_fixture( status = "inconclusive" elif frontier_stage["outcome"] == "difference": status = ( - expectation["classification"] - if expectation["classification"] != "match" + expectation["relationship"] + if expectation["relationship"] != "match" else "regression" ) inconclusive_reason = next( @@ -831,14 +741,11 @@ def compare_compiler_fixture( "fixture": fixture_id, "category": metadata.get("category", "unknown"), "status": status, - "expectedClassification": expectation["classification"], + "expectedRelationship": expectation["relationship"], "expectedNativeStatus": expected_status, "referenceStatus": oracle["compile"]["status"], "referenceGap": oracle["compile"]["status"] != native_status, - "evidence": expectation["evidence"], - "owner": expectation["owner"], "comparison": contract, - "note": expectation["note"], "reason": inconclusive_reason, "regressionStages": [ item["name"] for item in stages if item["outcome"] == "regression" @@ -914,7 +821,7 @@ def run( allow_inconclusive: bool, ) -> int: all_ids = fixture_ids(fixtures_root) - expectations = load_expectations(DEFAULT_EXPECTATIONS) + expectations = load_expectations(fixtures_root) missing = sorted(set(all_ids) - set(expectations)) extra = sorted(set(expectations) - set(all_ids)) if missing or extra: @@ -960,12 +867,11 @@ def run_compiler( fixtures_root: Path, report_path: Path, results_root: Path, - expectations_path: Path = DEFAULT_COMPILER_EXPECTATIONS, selected_ids: set[str] | None = None, allow_inconclusive: bool = False, ) -> int: all_ids = fixture_ids(fixtures_root) - expectations = load_compiler_expectations(expectations_path) + expectations = load_compiler_expectations(fixtures_root) missing = sorted(set(all_ids) - set(expectations)) extra = sorted(set(expectations) - set(all_ids)) if missing or extra: diff --git a/tools/overpy/run_native.py b/tools/overpy/run_native.py index afa9aad1..b4927045 100644 --- a/tools/overpy/run_native.py +++ b/tools/overpy/run_native.py @@ -16,7 +16,6 @@ ROOT = Path(__file__).resolve().parents[2] FIXTURES = ROOT / "crates/opy-rs/tests/fixtures/corpus" -COMPILER_EXPECTATIONS = ROOT / "tools/overpy/compiler-expectations.json" DEFAULT_RESULTS = ROOT / "target" / "opy-rs-compiler-results" DEFAULT_REPORT = ROOT / "target" / "opy-rs-compiler-report.json" @@ -48,7 +47,7 @@ def fixtures() -> list[tuple[Path, dict[str, Any]]]: return found -def run_semantic_evidence( +def run_semantic_comparison( binary: Path, directory: Path, metadata: dict[str, Any], @@ -76,20 +75,20 @@ def run_semantic_evidence( ) if completed.returncode != 0: raise NativeError( - f"{metadata['id']}: compatibility evidence failed " + f"{metadata['id']}: compatibility comparison failed " f"(exit {completed.returncode}): {completed.stderr.strip()}" ) try: result = json.loads(completed.stdout) except json.JSONDecodeError as error: raise NativeError( - f"{metadata['id']}: compatibility evidence did not produce JSON: " + f"{metadata['id']}: compatibility comparison did not produce JSON: " f"{completed.stderr.strip()}" ) from error if not isinstance(result, dict) or result.get("schemaVersion") != 1: - raise NativeError(f"{metadata['id']}: invalid compatibility evidence schema") + raise NativeError(f"{metadata['id']}: invalid compatibility comparison schema") if not isinstance(result.get("semanticWIR"), dict): - raise NativeError(f"{metadata['id']}: semantic-WIR evidence is missing") + raise NativeError(f"{metadata['id']}: semantic-WIR comparison is missing") return result @@ -141,7 +140,7 @@ def run_fixture( expectation["comparison"] == "semantic-wir" and result["compile"]["status"] == "success" ): - result["compatibility"] = run_semantic_evidence( + result["compatibility"] = run_semantic_comparison( semantic_binary, directory, metadata, @@ -161,7 +160,7 @@ def main(argv: list[str] | None = None) -> int: parser.add_argument( "--semantic-binary", type=Path, - help="path to the internal compatibility evidence binary", + help="path to the internal compatibility comparison binary", ) parser.add_argument("--results", type=Path, default=DEFAULT_RESULTS) parser.add_argument("--report", type=Path, default=DEFAULT_REPORT) @@ -180,12 +179,12 @@ def main(argv: list[str] | None = None) -> int: else binary.with_name("opy-compat") ) if not semantic_binary.is_file(): - raise NativeError(f"compatibility evidence binary does not exist: {semantic_binary}") + raise NativeError(f"compatibility comparison binary does not exist: {semantic_binary}") results_root = args.results.resolve() report_path = args.report.resolve() results_root.mkdir(parents=True, exist_ok=True) try: - expectations = diff.load_compiler_expectations(COMPILER_EXPECTATIONS) + expectations = diff.load_compiler_expectations(FIXTURES) except diff.DiffError as error: raise NativeError(str(error)) from error @@ -206,7 +205,6 @@ def main(argv: list[str] | None = None) -> int: FIXTURES, report_path, results_root, - COMPILER_EXPECTATIONS, allow_inconclusive=args.allow_inconclusive, ) except diff.DiffError as error: diff --git a/tools/overpy/run_oracle.py b/tools/overpy/run_oracle.py index 8ddc78f0..6dcd7b72 100644 --- a/tools/overpy/run_oracle.py +++ b/tools/overpy/run_oracle.py @@ -94,27 +94,26 @@ def _inside(path: Path, directory: Path) -> bool: def validate_fixture(path: Path) -> tuple[Path, dict[str, Any]]: metadata = load_json(path) - required = ("schemaVersion", "id", "category", "source", "expectedStatus", "provenance") + required = ("schemaVersion", "id", "category", "source") missing = [key for key in required if key not in metadata] if missing: raise RunnerError(f"{path}: missing fields: {', '.join(missing)}") if metadata["schemaVersion"] != 1: raise RunnerError(f"{path}: unsupported schemaVersion {metadata['schemaVersion']!r}") - if metadata["expectedStatus"] not in ("success", "failure"): - raise RunnerError(f"{path}: expectedStatus must be success or failure") if not isinstance(metadata["id"], str) or not metadata["id"]: raise RunnerError(f"{path}: id must be a non-empty string") if not isinstance(metadata["category"], str) or not metadata["category"]: raise RunnerError(f"{path}: category must be a non-empty string") - provenance = metadata["provenance"] - if not isinstance(provenance, dict): - raise RunnerError(f"{path}: provenance must be an object") - for key in ("kind", "origin", "license", "redistributable"): - if key not in provenance: - raise RunnerError(f"{path}: provenance missing {key}") - if not isinstance(provenance["redistributable"], bool): - raise RunnerError(f"{path}: provenance.redistributable must be boolean") + attribution = metadata.get("attribution") + if attribution is not None: + if not isinstance(attribution, dict): + raise RunnerError(f"{path}: attribution must be an object") + for key in ("kind", "origin", "license", "redistributable"): + if key not in attribution: + raise RunnerError(f"{path}: attribution missing {key}") + if not isinstance(attribution["redistributable"], bool): + raise RunnerError(f"{path}: attribution.redistributable must be boolean") fixture_dir = path.parent.resolve() source_value = metadata["source"] @@ -268,12 +267,7 @@ def run( continue try: actual = run_fixture(fixture_path, fixture, oracle_dir, oracle) - expected_status = fixture["expectedStatus"] actual_status = actual["compile"]["status"] - if actual_status != expected_status: - raise RunnerError( - f"expected {expected_status}, oracle returned {actual_status}" - ) snapshot_path = fixture_path.parent / SNAPSHOT_NAME if update: diff --git a/tools/overpy/tests/test_diff.py b/tools/overpy/tests/test_diff.py index 3df35298..8fb5ff7e 100644 --- a/tools/overpy/tests/test_diff.py +++ b/tools/overpy/tests/test_diff.py @@ -25,15 +25,16 @@ def setUpClass(cls): ) cls.oracle = json.loads(snapshot.read_text(encoding="utf-8")) - def test_expectations_cover_every_fixture_with_evidence(self): + def test_source_expectations_cover_every_fixture(self): expectations = diff.load_expectations() fixtures = set(diff.fixture_ids(CORPUS_DIR)) self.assertEqual(set(expectations), fixtures) for fixture, expectation in expectations.items(): - self.assertTrue(expectation["evidence"], fixture) - self.assertTrue(expectation["note"], fixture) + self.assertIn(expectation["relationship"], diff.EXPECTED_CLASSIFICATIONS) + self.assertNotIn("evidence", expectation) + self.assertNotIn("note", expectation) - def test_compiler_expectations_are_separate_and_cover_every_fixture(self): + def test_compiler_expectations_cover_every_fixture(self): source = diff.load_expectations() compiler = diff.load_compiler_expectations() fixtures = set(diff.fixture_ids(CORPUS_DIR)) @@ -41,9 +42,10 @@ def test_compiler_expectations_are_separate_and_cover_every_fixture(self): self.assertNotIn("comparison", source["synthetic/basic-rule"]) for fixture, expectation in compiler.items(): self.assertIn(expectation["comparison"], diff.EXPECTED_COMPILER_COMPARISONS) - self.assertTrue(expectation["evidence"], fixture) - self.assertTrue(expectation["owner"], fixture) - self.assertTrue(expectation["note"], fixture) + self.assertIn(expectation["relationship"], diff.EXPECTED_CLASSIFICATIONS) + self.assertNotIn("evidence", expectation) + self.assertNotIn("owner", expectation) + self.assertNotIn("note", expectation) def write_result(self, root: Path, result: dict): path = root / result["fixture"] / "result.json" @@ -299,13 +301,13 @@ def test_compiler_input_hash_mismatch_is_rejected(self): diff.load_compiler_expectations(), ) - def test_compile_result_rejects_public_semantic_wir_evidence(self): + def test_compile_result_rejects_public_semantic_wir_comparison(self): result = copy.deepcopy(self.oracle) result["compile"]["semanticWIR"] = {} with self.assertRaises(diff.DiffError): diff.require_result_shape(result, "compile result") - def test_compiler_semantic_wir_rejects_stale_direct_evidence(self): + def test_compiler_semantic_wir_rejects_stale_direct_comparison(self): fixture = "synthetic/expressions-values" oracle = json.loads( ( diff --git a/tools/overpy/tests/test_runner.py b/tools/overpy/tests/test_runner.py index 321fa240..69aa2e69 100644 --- a/tools/overpy/tests/test_runner.py +++ b/tools/overpy/tests/test_runner.py @@ -77,19 +77,38 @@ def test_repository_fixture_metadata_and_snapshots_are_valid(self): ] self.assertGreaterEqual(len(real_world), 6) for fixture in real_world: - self.assertTrue(fixture["provenance"]["redistributable"]) - self.assertEqual(fixture["provenance"]["modifications"], "none") + self.assertTrue(fixture["attribution"]["redistributable"]) + self.assertEqual(fixture["attribution"]["modifications"], "none") self.assertEqual( - len(fixture["provenance"]["sourceCommit"]), 40, + len(fixture["attribution"]["sourceCommit"]), 40, fixture["id"], ) overpy_cake = next( fixture for _, fixture in fixtures if fixture["id"] == "real-world/overpy-cake" ) - self.assertEqual(overpy_cake["provenance"]["kind"], "imported-example") + self.assertEqual(overpy_cake["attribution"]["kind"], "imported-example") - def test_real_world_gaps_have_minimized_provenance_linked_regressions(self): + def test_fixture_manifest_owns_concrete_test_declarations(self): + for fixture_path, fixture in run_oracle.discover_fixtures(CORPUS_DIR): + with self.subTest(fixture=fixture["id"]): + self.assertEqual(set(fixture["tests"]), {"source", "compiler"}) + attribution = fixture.get("attribution") + if attribution is not None: + self.assertIsInstance(attribution, dict) + self.assertTrue(attribution["origin"]) + self.assertTrue(attribution["license"]) + self.assertNotIn("expectedStatus", fixture) + self.assertNotIn("provenance", fixture) + self.assertNotIn("provenanceNote", fixture) + self.assertNotIn("runtimeSeconds", fixture) + self.assertNotIn("acquisitionMethod", fixture) + self.assertNotIn("evidencePolicy", fixture) + self.assertNotIn("evidence", fixture) + self.assertNotIn("note", fixture) + self.assertNotIn("owner", fixture) + + def test_real_world_gaps_have_minimized_regressions(self): fixtures = run_oracle.discover_fixtures(CORPUS_DIR) gaps = [ fixture @@ -112,11 +131,8 @@ def test_real_world_gaps_have_minimized_provenance_linked_regressions(self): self.assertGreaterEqual(len(regressions), 1, fixture["id"]) for regression in regressions: source = (CORPUS_DIR / fixture["id"] / regression["source"]).resolve() - self.assertTrue(source.is_file(), regression["id"]) - self.assertEqual(regression["derivedFrom"], fixture["source"]) - self.assertEqual(regression["expectedReferenceStatus"], fixture["expectedStatus"]) - self.assertEqual(regression["kind"], "minimized-regression") - self.assertIn("oracle:", regression["provenance"]) + self.assertTrue(source.is_file(), regression["source"]) + self.assertEqual(set(regression), {"source"}) def test_census_uses_opaque_workshop_owned_feature_ids(self): _, census = next( @@ -125,7 +141,6 @@ def test_census_uses_opaque_workshop_owned_feature_ids(self): if fixture[1]["id"] == "census/workshop-feature-census" ) self.assertEqual(census["censusContract"]["owner"], "workshop-rs") - self.assertEqual(census["censusContract"]["status"], "pending") self.assertTrue(census["workshopFeatureIds"]) self.assertTrue(all(isinstance(item, str) for item in census["workshopFeatureIds"]))