Skip to content

Add exhaustive category and per-feature Workshop semantic conformance tests #87

Description

@Teakowa

Goal

Add an exhaustive, data-driven Workshop conformance suite that verifies the audited native language surface by category and by individual feature, using deterministic offline input -> expected semantic/output behavior.

The suite should make it possible to answer both:

  • does an entire Workshop category remain conformant? and
  • which exact Action / Value / Event / Constant / operator / structural feature regressed?

Context

#19 established sharded census coverage for the repository's declared surface, but that census is derived from existing catalog/WIR capabilities. #86 adds the missing external documentation-driven inventory audit.

Once the documented inventory is authoritative, testing should stop relying mainly on representative examples and real-project incidental coverage. Every supported leaf capability should have an attributable conformance case where the language contract can be exercised offline.

This is a language/tooling correctness suite, not a live-game integration suite.

Scope

  • Build category-level conformance sets for at least:
    • Actions;
    • Values;
    • Events;
    • Constants / enum domains and members;
    • operators and variable-modification operations;
    • structure/control flow;
    • settings;
    • strings/localization where applicable.
  • Give every individual documented feature a stable, human-readable test identity, for example:
    • actions/damage;
    • values/raise-to-power;
    • events/player-dealt-damage;
    • constants/team/team-1;
    • operations/raise-to-power.
  • Derive the required coverage set from the audited native Workshop inventory established by Audit the complete documented Workshop native surface against workshop-rs #86 / docs/language-support.md, not only from the implementation catalog under test.
  • Prefer data-driven cases/generators over hundreds of unrelated hand-written Rust test functions.
  • For each supported capability, test the offline language contract that is applicable to that feature, including as relevant:
    • documented source spelling/category resolves to the expected canonical identity;
    • documented parameter count/order is accepted;
    • documented parameter types/domains and defaults are validated;
    • documented Value return contract is represented;
    • a minimal valid Workshop input parses/validates successfully;
    • representative invalid arity/domain/category input is rejected with the expected structured diagnostic class;
    • canonical emission uses the expected Workshop identity/spelling and argument ordering;
    • emitted output reparses to the same semantics;
    • supported locale conversion preserves the same canonical semantics.
  • Where deterministic presentation is part of the declared emitter contract, test normalized/canonical output presentation. Do not turn irrelevant whitespace or upstream formatter identity into a compatibility requirement.
  • Produce category summaries suitable for CI diagnostics, for example Actions, Values, Events, and Constants, while retaining exact per-feature failure identities.
  • Ensure the suite fails when an audited ✅ Supported feature has no conformance case, so support documentation and test coverage cannot silently diverge.
  • Keep 🚧 Coming soon and ❌ Unsupported entries explicit without fabricating passing tests for behavior that is not implemented.

Correctness boundary

The test oracle is the reviewed Workshop language/semantic contract and repository-declared canonical behavior.

The suite should verify that known inputs produce the expected parse/validation/semantic/emission results and that the documented support matrix, semantic model, and observable compiler/tooling behavior remain consistent.

No test should require execution inside Overwatch. Gameplay consequences such as actual damage dealt, movement, physics, timing, hero state, or engine-side effects are outside this suite because workshop-rs cannot integration-test them reliably.

A documented runtime note should only affect this suite when it constrains an offline semantic operation such as validation, static analysis, constant folding, or a source transformation. The suite must not introduce a game simulator to test such notes.

Non-goals

  • Live client/game integration tests.
  • Simulating Workshop runtime execution or gameplay effects.
  • Byte-identical comparison with OverPy, OSTW, or Blizzard-generated formatting.
  • Treating a large all-features fixture as a substitute for individually attributable cases.
  • Fixed-count assertions such as 219/219 that can stay green while the external inventory changes; completeness must be derived from the audited feature inventory.
  • Duplicating OPY/DEL source-language test suites.
  • Weakening validation or changing expected output merely to make the suite pass.

Acceptance criteria

  • Every audited ✅ Supported Workshop leaf capability that can be exercised offline has an attributable conformance case.
  • CI can report pass/fail by major category and exact feature identity.
  • Actions, Values, Events, and Constants each have dedicated category coverage rather than being represented only incidentally by corpus projects.
  • Same-name features in different semantic categories have independent cases; values/raise-to-power and the variable-modification Raise To Power operation cannot satisfy each other's coverage.
  • Valid test inputs resolve/validate/emit to the expected semantic result and supported canonical presentation.
  • Representative invalid inputs verify documented arity/domain/category rejection where applicable.
  • Supported emit -> parse round trips preserve canonical semantics.
  • Coverage fails if a supported documented feature is omitted from the suite.
  • No acceptance test depends on Overwatch client execution or claims gameplay/runtime correctness.

Dependencies / relationships

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions