[testing] Track the production-grade application testing platform

[kang-heewon]

Goal

Make Croco applications safe to change without requiring each team or coding agent to invent a separate testing architecture.

The target is not a custom test runner or a larger collection of mocks. Croco should provide production-parity isolation, deterministic failure scenarios, typed real-resource lifecycle, browser defaults, and executable evidence that explains which behavioral guarantees are still missing.

Delivery order

Foundation

• [testing] Introduce a production-parity TestKernel with explicit fidelity #1482 — production-parity TestKernel and explicit fidelity

This is the critical path. Higher-level evidence is not trustworthy while tests can silently use a testing-only composition path or process-global state.

Reliable application test runtime

• [testing] Add deterministic test runtime controls and leak detection #1483 — deterministic clock, IDs, environment/network policy, and leak detection
• [testing] Add typed test resources with explicit database isolation modes #1484 — typed PostgreSQL/Redis resources with rollback/commit/migration isolation
• [testing] Generate frontend component and browser journey testing defaults #1485 — generated frontend component and Playwright journey defaults
• [testing] Generalize failure drills into deterministic application scenarios #1486 — deterministic application scenarios generalized from failure drills

Executable assurance

• [testing] Emit unified executable test evidence across runners #1487 — unified test evidence across Vitest, Playwright, conformance, resources, and drills
• [testing] Build an Executable Assurance Graph from Croco contracts #1488 — Executable Assurance Graph derived from existing Croco artifacts
• [testing] Add contract-guided fuzzing and cross-runtime differential suites #1489 — contract-guided fuzzing and cross-runtime differential suites

Safe optimization

• [testing] Plan changed tests from contract and dependency impact #1490 — changed-test planning in shadow mode before selective execution is trusted

Architectural constraints

• Keep Vitest and Playwright as the runners; do not build a Croco runner.
• Keep Testcontainers and browser dependencies outside runner-neutral @croco/testing.
• Reuse contract graph, Problem registry, project/DI maps, provider profile, runtime capability, certification, and failure-drill artifacts instead of adding another manual source of truth.
• Never allow low-fidelity evidence to satisfy a higher-fidelity obligation.
• Do not enable changed-test-only PR gating until shadow runs demonstrate no unexplained selection misses over a documented observation window.
• Preserve deterministic machine-readable artifacts and actionable recovery commands for humans and coding agents.

Milestone outcomes

Milestone A — trustworthy harness

A generated application can run two concurrent production-parity kernels in one process without shared state, use deterministic runtime controls, and prove real commit semantics against a local PostgreSQL resource.

Milestone B — full-stack default

A newly generated frontend-bearing application includes component tests and a Playwright success/failure journey, with traces and backend evidence retained on failure.

Milestone C — assurance report

A contract or Problem change produces a report showing the required executable evidence, the evidence actually observed, its fidelity, and any missing or stale cases.

Milestone D — impact planning

croco test plan --changed explains selected tests and fallbacks, while shadow mode measures every selection miss before optimization is permitted.

Success criteria

• A developer can generate a Croco app and run unit/application/resource/component/journey checks without assembling a bespoke test stack.
• Production bootstrap, runtime target, dependency fidelity, and transaction isolation are explicit in test evidence.
• Retry, duplicate delivery, commit/response-loss, timeout, and telemetry failures are reproducible from a stable scenario ID and seed.
• CI reports missing behavioral evidence rather than relying only on command success or line coverage.
• AI coding agents receive stable contract IDs, source locations, replay commands, and recovery actions.

Non-goals

• Mandatory remote SaaS testing services.
• Supporting every provider/database/browser in the first release.
• A single test-quality score.
• Replacing package tests, existing conformance suites, or current release verification profiles.