Skip to content

[testing] Add contract-guided fuzzing and cross-runtime differential suites #1489

Description

@kang-heewon

Priority

P2 — depends on #1482, #1483, and evidence support from #1487.

Problem

Croco has typed route schemas, stable Problems, runtime capability manifests, and multiple runtime adapters, but it does not systematically use those artifacts to generate hostile inputs or compare runtime behavior. Handwritten examples miss malformed transport values, serialization boundaries, undocumented statuses, secret reflection, and Node/Lambda/Cloudflare differences.

Desired outcome

Add bounded, deterministic contract-guided fuzzing and cross-runtime differential suites using existing runners and property-testing libraries rather than a Croco-owned fuzz engine.

Scope

Contract-guided fuzzing

Generate valid and invalid inputs from supported route/schema metadata and verify invariants:

  • every response matches a declared success or Problem contract;
  • malformed transport input does not become an undocumented raw 500;
  • returned status and Problem code are declared;
  • canary secrets are not reflected in Problem, log, span, or serialized output;
  • failures retain seed, shrink path, and minimal counterexample.

Runtime differential

Execute the same request/scenario corpus against Node, Lambda, and Cloudflare adapters and compare canonical behavior. Differences are allowed only when represented by runtime capability policy.

Acceptance criteria

  • Support a documented subset of Zod/contract constructs and fail clearly for unsupported generation shapes.
  • PR runs use a bounded deterministic case budget; larger nightly/manual profiles are separate.
  • Failed cases persist seed, shrink path, minimal input, runtime, and replay command.
  • Fuzzing checks declared response unions rather than assuming every schema-valid input must succeed.
  • Differential comparison covers status, response/Problem schema, stable headers, trace propagation, and declared runtime lifecycle outcomes.
  • Runtime-specific streaming, deadline, abort, waitUntil, flush, and shutdown behavior is checked against capability declarations rather than forced identical.
  • An undeclared runtime difference is blocking in the dedicated suite.
  • Generated-app smoke includes at least one bounded route fuzz case and one Node/Lambda parity case.
  • No remote provider credentials or public network are required.

Non-goals

  • Generating arbitrary domain-valid database states automatically.
  • Treating property tests as replacements for explicit business examples.
  • Guaranteeing complete schema support in the first version.

Validation

Seed regressions demonstrating malformed input, an undocumented Problem/status, secret reflection detection, minimized counterexamples, and an intentionally introduced runtime mismatch.

Metadata

Metadata

Assignees

No one assigned

    Labels

    contractsquality-gateruntimeRuntime behavior, lifecycle, and environment capability worktestingTesting utilities, harnesses, and verification workflows

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions