feat: run deterministic application failure scenarios - #1793
Merged
Conversation
|
Warning Review limit reached
Next review available in: 32 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (9)
📝 WalkthroughWalkthrough
Changes시나리오 런타임
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant SaaSScenario
participant ScenarioRuntime
participant DemoBillingGateway
participant ScenarioReport
SaaSScenario->>ScenarioRuntime: run checkout response-loss scenario
ScenarioRuntime->>DemoBillingGateway: execute first checkout with lost response
DemoBillingGateway-->>ScenarioRuntime: reuse checkout on retry
SaaSScenario->>DemoBillingGateway: replay the same checkout request
DemoBillingGateway-->>ScenarioReport: return the same checkout and creation count
ScenarioRuntime->>ScenarioReport: record evidence and validate expectations
Possibly related PRs
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a6c00448f6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/docs/src/content/docs/api/testing/src/classes/ScenarioContractProblem.md`:
- Line 8: Update the source JSDoc for ScenarioContractProblem to describe it as
the concrete Problem subclass for the testing/scenario-contract-invalid problem
code and scenario contract validation failures, rather than the generic RFC 7807
base class. Regenerate TypeDoc so ScenarioContractProblem.md reflects the
corrected description.
In `@packages/testing/src/tests/ScenarioRuntime.spec.ts`:
- Line 63: Replace the invalid toHaveLength assertions with size-based
assertions for the Map/Set collections: update
packages/testing/src/tests/ScenarioRuntime.spec.ts lines 63-63 for committed and
lines 93-93 for handled to assert each collection’s size equals 1.
- Around line 154-167: Update the second replay test around runtime.execute so
it explicitly asserts that execute rejects, using the rejects-based assertion
while preserving the existing Problem type and shape checks on the captured
error. Ensure the test fails when replay.run completes without throwing.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 5d08be5d-5ecf-4751-8e49-e2bb973ad620
⛔ Files ignored due to path filters (1)
packages/problems-core/src/generated/problem-code-registry.tsis excluded by!**/generated/**
📒 Files selected for processing (42)
.changeset/replayable-application-scenarios.mddocs/problem-code-registry.jsonpackages/create-croco-app/src/tests/templates-build.spec.tspackages/create-croco-app/templates/saas/apps/api-server/src/demo/failure-drill-smoke.tspackages/create-croco-app/templates/saas/apps/api-server/src/saasDemo.tspackages/docs/src/content/docs/api/problems-core/src/classes/Problem.mdpackages/docs/src/content/docs/api/problems-core/src/variables/CROCO_PROBLEM_CODE_REGISTRY.mdpackages/docs/src/content/docs/api/testing/src/classes/ScenarioContractProblem.mdpackages/docs/src/content/docs/api/testing/src/classes/ScenarioRuntime.mdpackages/docs/src/content/docs/api/testing/src/functions/createScenarioRuntime.mdpackages/docs/src/content/docs/api/testing/src/functions/duplicateDelivery.mdpackages/docs/src/content/docs/api/testing/src/functions/failExporter.mdpackages/docs/src/content/docs/api/testing/src/functions/interruptProcess.mdpackages/docs/src/content/docs/api/testing/src/functions/loseResponse.mdpackages/docs/src/content/docs/api/testing/src/functions/replayScenarioRuntime.mdpackages/docs/src/content/docs/api/testing/src/functions/retryableFailure.mdpackages/docs/src/content/docs/api/testing/src/functions/serializeScenarioReport.mdpackages/docs/src/content/docs/api/testing/src/functions/terminalFailure.mdpackages/docs/src/content/docs/api/testing/src/functions/timeout.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioBoundary.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioDuplicateDelivery.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioEvidence.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioEvidenceKind.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioExpectation.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioFailure.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioFailureKind.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioProblemFailure.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioReplayDuplicateDelivery.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioReplayFailure.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioReplayMetadata.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioReplayProblemDetails.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioReplayProblemFailure.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioReport.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioRuntimeOptions.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioTimelineEntry.mdpackages/docs/src/content/docs/api/testing/src/variables/SCENARIO_REPORT_SCHEMA_VERSION.mdpackages/docs/src/content/docs/en/reference/problem-recovery-cookbook.mdpackages/testing/README.mdpackages/testing/src/index.tspackages/testing/src/libs/ScenarioRuntime.tspackages/testing/src/tests/ScenarioRuntime.spec.tspublic-api-surface.snapshot.json
kang-heewon
force-pushed
the
feat/1486-deterministic-scenario-runtime
branch
from
August 9, 2026 09:06
a6c0044 to
2ac6a70
Compare
📊 Benchmark Results✅ All benchmarks passed
Updated: 2026-08-09T13:38:23.619Z · Commit: c5af5aa |
kang-heewon
force-pushed
the
feat/1486-deterministic-scenario-runtime
branch
from
August 9, 2026 09:28
2ac6a70 to
7d21177
Compare
kang-heewon
force-pushed
the
feat/1486-deterministic-scenario-runtime
branch
from
August 9, 2026 10:56
7d21177 to
9d60ea6
Compare
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/problem-code-registry.json`:
- Around line 2075-2104: Update the recovery metadata for
billing/checkout-idempotency-drift so it explicitly describes an
already-detected idempotency violation after retries, uses a non-retryable
retryability value, and replaces the generic retry-oriented userAction with
guidance appropriate for investigation and reconciliation. Keep the problem
code, category, status, telemetry, lifecycle, and source metadata unchanged.
In `@packages/testing/src/libs/ScenarioRuntime.ts`:
- Around line 8-61: Unify the boundary and failure-kind definitions by declaring
SCENARIO_BOUNDARIES and SCENARIO_FAILURE_KINDS with as const, then derive
ScenarioBoundary and ScenarioFailureKind from each array’s element type using
indexed access. Remove the duplicated string-literal unions while preserving the
existing values and readonly behavior.
- Around line 334-363: Prevent repeated invocation of ScenarioRuntime.run by
tracking whether the instance has already started or completed a run and
throwing ScenarioContractProblem on any subsequent call. Ensure the guard
executes before the existing scenario execution and report-validation flow so
evidence, problems, and timeline cannot accumulate across runs.
- Around line 540-552: Update assertReplayMetadata to accept unknown and give it
an assertion signature that narrows the value to ScenarioReplayMetadata after
validation. Also change replayScenarioRuntime’s input to unknown and rely on the
post-assertion narrowed replay value, preserving the existing runtime checks and
parsed JSON flow.
- Around line 365-380: Update executeOperation so failures from
recordProblem(error), including normalizeProblemDetails failures, are caught and
linked to the original Problem as its cause while preserving the original
Problem as the error ultimately thrown. Ensure reporting failure does not
prevent the original boundary Problem from being propagated.
- Around line 645-693: Update assertPlainJsonShape and normalizeJsonValue so
undefined is allowed only as an object property value: skip validation for such
properties and omit their keys from normalized objects. Continue rejecting
undefined nested within retained values, including array elements and standalone
inputs, while preserving conditional empty objects such as Problem.toJSON()
results.
In `@packages/testing/src/tests/ScenarioRuntime.spec.ts`:
- Around line 4-17: Separate ScenarioReplayMetadata from the value import in
ScenarioRuntime.spec.ts and import it through a dedicated import type
declaration. Keep all runtime symbols in the existing import from ../index
unchanged.
- Around line 303-305: Replace the TypeError thrown by the timedStep validation
in ScenarioRuntime.spec.ts with ScenarioContractProblem, preserving the existing
invalid-fixture condition and message while supplying the appropriate stable
diagnostic code required by the Problem contract.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 54c7af1b-8f5a-4ba9-91cb-691a8e35509a
⛔ Files ignored due to path filters (1)
packages/problems-core/src/generated/problem-code-registry.tsis excluded by!**/generated/**
📒 Files selected for processing (14)
docs/problem-code-registry.jsonpackages/docs/src/content/docs/api/framework-context/src/classes/RuntimeInspectorConfigurationProblem.mdpackages/docs/src/content/docs/api/framework-context/src/type-aliases/RuntimeInspectorOptions.mdpackages/docs/src/content/docs/api/integrations-posthog/src/variables/POSTHOG_CONFIG_TOKEN.mdpackages/docs/src/content/docs/api/problems-core/src/classes/Problem.mdpackages/docs/src/content/docs/api/problems-core/src/variables/CROCO_PROBLEM_CODE_REGISTRY.mdpackages/docs/src/content/docs/api/testing/src/classes/ScenarioContractProblem.mdpackages/docs/src/content/docs/api/transports-graphql/src/classes/GraphQLRequestHandlingFailedProblem.mdpackages/docs/src/content/docs/api/transports-http/src/classes/DiagnosticsConfigurationProblem.mdpackages/docs/src/content/docs/api/transports-http/src/type-aliases/DiagnosticsConfigurationProblemOptions.mdpackages/docs/src/content/docs/en/reference/problem-recovery-cookbook.mdpackages/testing/src/libs/ScenarioRuntime.tspackages/testing/src/tests/ScenarioRuntime.spec.tspublic-api-surface.snapshot.json
kang-heewon
force-pushed
the
feat/1486-deterministic-scenario-runtime
branch
from
August 9, 2026 11:24
9d60ea6 to
bc2caa6
Compare
kang-heewon
force-pushed
the
feat/1486-deterministic-scenario-runtime
branch
from
August 9, 2026 11:57
bc2caa6 to
2ccacb5
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/testing/src/libs/ScenarioRuntime.ts`:
- Around line 725-734: Update durationToMilliseconds to validate numeric
durations before returning them: require a finite, non-negative integer, and
throw ScenarioContractProblem for invalid values such as negative, fractional,
or NaN inputs. Apply the same constraint to parsed string durations so the
values passed to toReplayFailure and advanceBy always satisfy replay metadata
requirements.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 4fe345a3-70fa-4b5c-aa43-0e2a3c73129e
⛔ Files ignored due to path filters (1)
packages/problems-core/src/generated/problem-code-registry.tsis excluded by!**/generated/**
📒 Files selected for processing (15)
docs/problem-code-registry.jsonpackages/create-croco-app/src/tests/templates-build.spec.tspackages/docs/src/content/docs/api/problems-core/src/classes/Problem.mdpackages/docs/src/content/docs/api/problems-core/src/variables/CROCO_PROBLEM_CODE_REGISTRY.mdpackages/docs/src/content/docs/api/protocols-core/src/variables/PROJECT_MANIFEST_BUNDLE_ARTIFACTS.mdpackages/docs/src/content/docs/api/telemetry-sdk-node/src/classes/TelemetryInitializationConflictProblem.mdpackages/docs/src/content/docs/api/testing/src/classes/ScenarioRuntime.mdpackages/docs/src/content/docs/api/testing/src/functions/replayScenarioRuntime.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioBoundary.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/ScenarioFailureKind.mdpackages/docs/src/content/docs/en/reference/problem-recovery-cookbook.mdpackages/testing/src/libs/ScenarioRuntime.tspackages/testing/src/tests/ScenarioRuntime.spec.tspublic-api-surface.snapshot.jsonscripts/problem-registry.mts
💤 Files with no reviewable changes (1)
- packages/docs/src/content/docs/en/reference/problem-recovery-cookbook.md
kang-heewon
force-pushed
the
feat/1486-deterministic-scenario-runtime
branch
from
August 9, 2026 12:20
2ccacb5 to
a2ddeb7
Compare
Member
Author
|
@coderabbitai resolve |
✅ Action performedComments resolved and changes approved. |
kang-heewon
force-pushed
the
feat/1486-deterministic-scenario-runtime
branch
from
August 9, 2026 13:28
a2ddeb7 to
17aebf6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Outcome
Applications can now compose ordered, named failure timelines with a runner-neutral
ScenarioRuntimeacross transaction, event, task, trigger, provider, retry, and telemetry boundaries. The runtime provides virtual time, deterministic replay metadata, stable versioned reports, strict Problem and evidence multiplicity assertions, and explicit injection for duplicate delivery, response loss, timeout, retryable or terminal failure, process interruption, and exporter failure.Generated SaaS applications extend the existing operational failure catalog with a real billing-gateway scenario that commits checkout creation, loses the response, retries with the same idempotency key, and proves exactly one checkout was created.
Fixes #1486
Reviewer context
ScenarioRuntimeis additive.@croco/testing,create-croco-app, and the generated@croco/problems-coreregistry are synchronized.Verification
pnpm --dir packages/testing test— 9 files and 175 tests passedCROCO_GENERATED_SMOKE_CASES=goal-saas-api pnpm create-croco-app:smoke— generated app lint, contracts, doctor, typecheck, build, tests, demo flow, and failure drill passedpnpm strict-contract-typecheck— no added diagnostics; 503 accepted baseline diagnostics matchedpnpm public-api:check— 115 package snapshots matchedpnpm problem-registry:check— 591/591 codes matchedpnpm docs:api-triggers:check— passedpnpm changeset-required:check— all affected publishable packages coveredpnpm check— 24/25 repository gates passed; 1 not applicablegit diff --check— passedThe full API-doc drift command regenerated and built all documentation successfully, then reported only pre-existing unrelated drift for RuntimeInspector and HTTP diagnostics pages on trunk; no testing API page drift remained.
Review gates
TestRuntimeand existing evidence redaction, keeps failure behavior at explicit boundary wrappers, and adds one application-backed template scenario without replacing the operational catalog.Residual risk
Scenario replay reproduces the deterministic failure schedule and controls, not arbitrary external system state or process checkpoints; those remain explicit non-goals of this API.
Summary by CodeRabbit
새로운 기능
문서