[testing] Generalize failure drills into deterministic application scenarios

[kang-heewon]

Priority

P1 — depends on #1482 and should use deterministic controls from #1483.

Problem

Croco already models important operational failures through provider and operational failure drills, but the catalog is primarily fixed and provider-oriented. Applications cannot easily compose ordered failures such as commit success followed by response loss, duplicate webhook delivery, task retry, and telemetry flush failure while preserving deterministic replay evidence.

Desired outcome

Generalize the existing failure-drill primitives into a runner-neutral ScenarioRuntime for application-specific success and failure timelines without adding test-only branches to production code.

Illustrative API:

scenario
  .at("billing.checkout", provider.accept().loseResponse())
  .at("webhook.receive", duplicate(2))
  .at("task.publish", failOnce(retryable("qstash/timeout")))
  .expect(problem("billing/checkout-pending"))
  .expectEventOnce("subscription.activated");

Acceptance criteria

• Existing failure-drill APIs remain compatible or have a documented migration path.
• Scenarios support ordered steps, named failure points, virtual-time advancement, and deterministic replay metadata.
• Failure injection wraps Croco boundaries such as transaction, event, task, trigger, provider, retry, and telemetry rather than adding NODE_ENV=test branches to production logic.
• Scenarios can model duplicate delivery, lost response, timeout, retryable/terminal provider failure, process-boundary interruption, and exporter/flush failure.
• Assertions cover Problem/diagnostic outcomes, recovery actions, event/task multiplicity, audit evidence, and telemetry evidence.
• Scenario reports are timestamp-stable and serializable as a versioned artifact.
• A minimal failed scenario can be replayed by scenario ID, seed, and timeline.
• Generated SaaS applications include one app-backed scenario that extends rather than replaces the existing operational failure catalog.
• Tests prove idempotency under a commit-success/response-loss/retry sequence and duplicate webhook delivery.

Non-goals

• A general distributed-systems simulator.
• Arbitrary Node process checkpoint/restore.
• Replacing live provider smoke tests.

Validation

Add scenarios spanning transaction, outbox/event publication, task retry, webhook duplication, and telemetry failure, with identical reports across repeated seeded runs.