feat: emit unified executable test evidence - #1671
Merged
Merged
Conversation
|
Warning Review limit reached
Next review available in: 10 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (83)
📝 WalkthroughWalkthrough테스트 증거 v1 계약과 JSON Schema를 추가했습니다. Vitest·Playwright 리포터가 공통 레코드를 생성합니다. CI는 JSON·Markdown 번들을 만들고, 검증 프로필은 조건에 맞는 증거를 재사용합니다. Changes테스트 증거 계약과 핵심 엔진
실행기 리포터와 출력
번들 및 검증 통합
문서 및 릴리스 지원
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant TestRunner
participant EvidenceReporter
participant EvidenceWriter
participant BundleCLI
participant Verification
TestRunner->>EvidenceReporter: 테스트 결과와 시도 전달
EvidenceReporter->>EvidenceWriter: TestEvidenceRecord 기록
EvidenceWriter->>BundleCLI: JSON 레코드 제공
BundleCLI->>BundleCLI: 레코드 검증 및 bundle.json 생성
BundleCLI->>Verification: 검증 증거 번들 제공
Verification->>Verification: 일치하는 artifact와 명령 확인
Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
📊 Benchmark Results✅ All benchmarks passed
Updated: 2026-08-01T22:44:43.897Z · Commit: 504db68 |
kang-heewon
force-pushed
the
issue-1487-unified-test-evidence
branch
3 times, most recently
from
August 1, 2026 17:13
c6208ed to
31f91c2
Compare
There was a problem hiding this comment.
Actionable comments posted: 20
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
scripts/verification-manifest.mts (1)
547-557: 🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
--force는 모든 실행에서 test 태스크 캐시를 무효화합니다.
--force는 영향 범위 내 모든test태스크를 항상 다시 실행합니다.test는 이 매니페스트에서 가장 긴 태스크입니다(timeout 45분). 따라서 CI와 로컬 검증의 캐시 재사용이 완전히 사라지고, 저장소의ci-performance-budget게이트에도 부담이 갑니다.근본 원인은
turbo.json이CROCO_TEST_EVIDENCE_DIR를globalPassThroughEnv에 넣은 점입니다. passthrough 환경 변수는 태스크 해시에 포함되지 않으므로, 캐시 적중 시 증거 파일이 생성되지 않습니다. 다음 중 하나를 사용하면 캐시를 유지하면서 증거를 보장할 수 있습니다.
test태스크의env에CROCO_TEST_EVIDENCE_DIR를 추가해 해시 입력으로 만듭니다.- 증거 파일 경로를
test태스크의outputs로 선언해 캐시 복원 시 파일이 복원되게 합니다.두 방식 모두
.github/workflows/ci.ymlLine 203-211의 fallback 단계와 함께 동작합니다.♻️ 제안 변경
command: [ "pnpm", "turbo", "run", "test", ...affectedArguments, - "--force", "--summarize", "--continue=always", ],
turbo.json의test태스크에 해시 입력을 추가합니다.{ "tasks": { "test": { "env": ["CROCO_TEST_EVIDENCE_DIR"] } } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/verification-manifest.mts` around lines 547 - 557, Remove the unconditional --force option from the test command in the verification manifest and update the test task configuration in turbo.json to include CROCO_TEST_EVIDENCE_DIR as a hashed env input, preserving cache reuse while ensuring evidence handling remains compatible with the CI fallback.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/docs/src/content/docs/api/testing/src/classes/TestEvidenceContractError.md`:
- Around line 8-16: Update the original TypeScript definition of
TestEvidenceContractError to extend the project’s RFC 7807 Problem base class
instead of Error, and update the writer path that throws this error to use the
Problem subclass. Do not edit the generated Markdown directly; regenerate the
API documentation and validate it with pnpm docs:api:check.
In `@packages/testing/README.md`:
- Around line 130-132: Update the evidence reuse description in the verification
profiles documentation to state that reuse requires exact matches for command,
profile, provenance, observed contract ID, metadata.commitSha, and all required
artifacts. Clarify that records failing any condition are not reused and the
command executes normally.
In `@packages/testing/src/libs/test-evidence-files.ts`:
- Around line 23-38: Update the returned evidence-writer callback around
outputPath and writeFileSync to avoid the existsSync/readFileSync/write race
when multiple processes share the directory: write serialized content to a
unique temporary file, then atomically publish it with renameSync while
preserving collision detection and throwing TestEvidenceContractError when
existing content differs. Verify the CI configuration uses a shared
CROCO_TEST_EVIDENCE_DIR so the implementation covers parallel package writers.
In `@packages/testing/src/libs/test-evidence-reporters.ts`:
- Around line 1-12: Separate the mixed imports in
packages/testing/src/libs/test-evidence-reporters.ts lines 1-12 by keeping
createTestEvidenceRecord in a value import and moving all TestEvidence* symbols
into a separate import type statement; likewise, in
packages/testing/src/libs/test-evidence-files.ts lines 5-9, keep
serializeTestEvidence and TestEvidenceContractError in the value import and move
TestEvidenceRecord into a separate import type statement, preserving the
required external, internal, then relative import ordering.
- Around line 53-67: Update the VitestTask.result and PlaywrightTestResult types
to expose their framework-provided failure fields (Vitest errors and Playwright
error), and use those fields in the default diagnostics paths around the
evidence reporter methods so failed tests include their actual errors. Preserve
custom diagnostics callbacks while giving them a typed context.source that
exposes the same failure information.
In `@packages/testing/src/libs/test-evidence.mts`:
- Around line 252-256: Update the non-record validation branch in the
test-evidence parser to throw a message identifying that the input shape is
invalid, rather than reporting a schemaVersion mismatch. Keep the
schemaVersion-specific message for record inputs whose version is incorrect,
using the existing TestEvidenceContractError and TEST_EVIDENCE_SCHEMA_VERSION
symbols.
- Around line 570-580: Update assertNoTestEvidenceSecrets so JSON.stringify
failures, including BigInt inputs, and non-string results such as undefined are
converted into TestEvidenceContractError rather than causing TypeError. Validate
the serialized result before calling includes, while preserving the existing
secret-sample detection behavior for successfully serialized strings.
- Around line 179-182: Reorder the logic around assertAttempts and
classifyTestEvidenceOutcome so the copied input is sorted by attempt before
validation. Then pass the sorted attempts to assertAttempts, preserving
rejection of missing or duplicate attempt values while accepting valid entries
supplied in any order.
- Around line 847-849: Update escapeMarkdown to escape pipe characters,
newlines, and backticks so multiline record IDs cannot break Markdown tables or
artifact lists. In the missing-artifact output near recordId/path, apply
escapeMarkdown to both values before interpolation, while preserving existing
formatting.
- Around line 153-171: Update TestEvidenceContractError and
TestEvidenceFidelityError to extend the Problem type from `@croco/problems-core`
instead of Error, while preserving their existing messages and names. Expose
each error’s diagnostic code through the established problemCodes mechanism or
an explicit diagnostic-code property, and associate the codes with the
appropriate problem category.
In `@public-api-surface.snapshot.json`:
- Around line 26130-26151: Update the asset export targets for
test-evidence-bundle-v1 and test-evidence-v1 in public-api-surface.snapshot.json
to use ./dist/schemas/test-evidence-bundle-v1.json and
./dist/schemas/test-evidence-v1.json, matching the files produced by
copy-schemas.mjs; leave the export paths unchanged.
- Around line 24746-24750: Regenerate public-api-surface.snapshot.json using the
standard public API snapshot generation process with the required public API
sources and build outputs available. Ensure the assertNoTestEvidenceSecrets
export from ./libs/test-evidence.mjs receives its inferred declarationKind,
consistent with the inference path in scripts/public-api-surface.mts.
In `@scripts/release-spine-evidence.mts`:
- Line 1054: Update the reuse path around collectArtifactReferences so it does
not pass startedMs = 0 and incorrectly mark all existing artifacts fresh.
Preserve reused artifacts as fresh: false and record the corresponding reuse
record ID in each artifact reference, while leaving normal current-run freshness
detection unchanged.
- Around line 1053-1069: Move the check.reusedEvidence branch in the
report-update flow to execute after the applicable === false branch, preserving
the not_applicable result for checks that are not applicable even when reusable
evidence matches. Remove the now-redundant later applicable === false block only
if the reordered flow retains its behavior, while keeping the reused-artifact
passed path unchanged.
- Around line 1208-1228: Update the test-evidence loading flow around
readFileSync, JSON.parse, and assertTestEvidenceBundle to catch file-read and
JSON-parse failures and convert them into VerificationProblem using the existing
input-error code. Remove the duplicated schema checks, retaining only the
passed-status and empty-missingArtifacts requirements before calling
assertTestEvidenceBundle. Remove the assertTestEvidenceRecord import if it
becomes unused.
In `@scripts/test-evidence-bundle.mts`:
- Around line 157-170: Preserve the original failure details in both
evidence-loading paths. In scripts/test-evidence-bundle.mts lines 157-170, catch
the error in the normalizeTestEvidenceInput flow and pass its message into
inputFailureRecord’s diagnostic. In scripts/release-spine-evidence.mts lines
1208-1228, wrap readFileSync, JSON.parse, and assertTestEvidenceBundle in
try/catch, then throw INVALID_TEST_EVIDENCE_INPUT VerificationProblem including
the underlying error message.
- Around line 157-170: Update the catch block surrounding
normalizeTestEvidenceInput in the test-evidence parsing flow to retain the
caught error and include its diagnostic message in the inputFailureRecord
evidence. Preserve the existing failure code and regeneration guidance while
ensuring both JSON parsing and schema-validation causes are traceable in the
bundle.
In `@scripts/tests/release-spine-evidence.spec.ts`:
- Around line 1016-1027: Split the combined reuseTestEvidence test into two
independent tests: one covering missing required artifacts and another covering
replay command mismatch. Move each setup and assertion into its corresponding
test, and remove the non-null assertion on value.records[0] by accessing the
record safely or restructuring the fixture.
- Around line 928-932: fixture JSON 리터럴에 profile: "spine"을 직접 포함하도록 수정하고, 이후
evidencePath를 다시 읽어 value.records[0]!.metadata.profile을 설정한 뒤 재작성하는 블록을 제거하십시오.
해당 변경으로 non-null assertion과 불필요한 파일 읽기·쓰기를 함께 없애고, fixture가 한 번에 올바른 내용을 생성하도록
유지하십시오.
In `@scripts/tests/test-evidence-bundle.spec.ts`:
- Around line 27-176: normalizeTestEvidenceInput의 미검증 입력 분기를 커버하도록 test evidence
번들 테스트를 확장하십시오. 기존 writeTestEvidenceBundle 테스트에 레코드 배열 입력과
croco.test-evidence/v1 번들 입력 사례를 각각 추가하고, reporter가 생성한 파일이 두 형식 모두 정상적으로 정규화되어
기대한 상태와 레코드로 출력되는지 검증하십시오.
---
Outside diff comments:
In `@scripts/verification-manifest.mts`:
- Around line 547-557: Remove the unconditional --force option from the test
command in the verification manifest and update the test task configuration in
turbo.json to include CROCO_TEST_EVIDENCE_DIR as a hashed env input, preserving
cache reuse while ensuring evidence handling remains compatible with the CI
fallback.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 995c9831-cf15-4a61-a848-bebed33d59e0
📒 Files selected for processing (69)
.changeset/unified-test-evidence.md.github/workflows/ci.ymlpackage.jsonpackages/docs/src/content/docs/api/testing/src/classes/CrocoPlaywrightEvidenceReporter.mdpackages/docs/src/content/docs/api/testing/src/classes/CrocoVitestEvidenceReporter.mdpackages/docs/src/content/docs/api/testing/src/classes/TestEvidenceContractError.mdpackages/docs/src/content/docs/api/testing/src/classes/TestEvidenceFidelityError.mdpackages/docs/src/content/docs/api/testing/src/functions/assertNoTestEvidenceSecrets.mdpackages/docs/src/content/docs/api/testing/src/functions/assertTestEvidenceBundle.mdpackages/docs/src/content/docs/api/testing/src/functions/assertTestEvidenceFidelity.mdpackages/docs/src/content/docs/api/testing/src/functions/assertTestEvidenceRecord.mdpackages/docs/src/content/docs/api/testing/src/functions/classifyTestEvidenceOutcome.mdpackages/docs/src/content/docs/api/testing/src/functions/createTestEvidenceBundle.mdpackages/docs/src/content/docs/api/testing/src/functions/createTestEvidenceFileWriter.mdpackages/docs/src/content/docs/api/testing/src/functions/createTestEvidenceRecord.mdpackages/docs/src/content/docs/api/testing/src/functions/createTestKernelEvidenceRecord.mdpackages/docs/src/content/docs/api/testing/src/functions/redactTestEvidence.mdpackages/docs/src/content/docs/api/testing/src/functions/renderTestEvidenceMarkdown.mdpackages/docs/src/content/docs/api/testing/src/functions/serializeTestEvidence.mdpackages/docs/src/content/docs/api/testing/src/functions/testEvidenceFidelityFromKernel.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceArtifactProbe.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceAttachment.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceAttempt.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceAttemptOutcome.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceBundle.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceDiagnostic.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceFidelity.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceFidelityRequirement.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceFileWriterOptions.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceIntent.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceJsonValue.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceMissingArtifact.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceObservation.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceOutcome.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceRecord.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceRecordInput.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceReplay.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceReporterContext.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceReporterOptions.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceResourceStatus.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceRunner.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceTiming.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestKernelEvidenceRecordInput.mdpackages/docs/src/content/docs/api/testing/src/variables/TEST_EVIDENCE_SCHEMA_VERSION.mdpackages/docs/src/content/docs/api/webhooks-core/src/classes/UnknownWebhookEventProblem.mdpackages/testing/README.mdpackages/testing/package.jsonpackages/testing/schemas/test-evidence-bundle-v1.schema.jsonpackages/testing/schemas/test-evidence-v1.schema.jsonpackages/testing/scripts/copy-schemas.mjspackages/testing/src/index.tspackages/testing/src/libs/test-evidence-files.tspackages/testing/src/libs/test-evidence-reporters.tspackages/testing/src/libs/test-evidence.mtspackages/testing/src/playwright-reporter.tspackages/testing/src/tests/TestEvidence.spec.tspackages/testing/src/vitest-reporter.tspackages/testing/vitest.config.tspublic-api-surface.snapshot.jsonscripts/release-spine-evidence.mtsscripts/test-evidence-bundle.mtsscripts/tests/ci-workflow.spec.tsscripts/tests/release-spine-evidence.spec.tsscripts/tests/test-evidence-bundle.spec.tsscripts/tests/turbo-task-contract.spec.tsscripts/tests/verification-manifest.spec.tsscripts/verification-manifest.mtsscripts/workflow-verification-contract.mtsturbo.json
kang-heewon
force-pushed
the
issue-1487-unified-test-evidence
branch
2 times, most recently
from
August 1, 2026 18:30
f19a6a4 to
17e57b6
Compare
kang-heewon
commented
Aug 1, 2026
There was a problem hiding this comment.
Actionable comments posted: 14
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
scripts/verification-manifest.mts (1)
543-558: 🚀 Performance & Scalability | 🔵 Trivial
--force는test작업의 Turbo 캐시 재사용을 항상 비활성화합니다.
--force를 추가하면spine/publish프로필에서test작업이 캐시 적중 여부와 무관하게 항상 다시 실행됩니다. 이는 Vitest reporter가 실제로 실행되어 신선한 증거를 생성하도록 보장하는 의도된 선택으로 보입니다. 다만 동일 커밋을 재검증하거나 영향받은 패키지가 이미 캐시되어 있는 경우에도 매번 전체 재실행 비용이 발생합니다. CI 실행 시간에 미치는 영향을 관찰하고, 필요하면 캐시된 결과에서 증거 파일만 복원하는 방식도 검토하십시오.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/verification-manifest.mts` around lines 543 - 558, Update the test command in the “test” manifest entry to avoid unconditionally passing Turbo’s “--force” flag, allowing cached test results to be reused; preserve the summarized-test behavior and ensure cached executions still restore the required evidence files when supported by the existing verification flow.
♻️ Duplicate comments (1)
packages/testing/src/libs/test-evidence-files.ts (1)
29-39: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win부분 쓰기로 인한 거짓 충돌 오류가 남아 있습니다.
flag: "wx"는 생성 경쟁을 제거합니다. 그러나writeFileSync는 원자적이지 않습니다. 두 프로세스가 동일한outputPath를 대상으로 할 때, 두 번째 프로세스는 EEXIST를 받고 Line 33에서 아직 완성되지 않은 파일을 읽을 수 있습니다. 이때 내용이 다르므로 실제 충돌이 없어도TestEvidenceContractError가 발생합니다. CI가 여러 패키지에서 같은CROCO_TEST_EVIDENCE_DIR를 공유하면 증거 수집이 간헐적으로 실패합니다.임시 파일에 쓴 뒤
renameSync로 게시하십시오. 이 방식은 게시를 원자적으로 만들고 충돌 검사도 완성된 내용만 비교합니다.🔒️ 제안된 수정
-import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";const serialized = serializeTestEvidence(record); + const temporaryPath = `${outputPath}.${process.pid}.tmp`; try { - writeFileSync(outputPath, serialized, { flag: "wx" }); + writeFileSync(temporaryPath, serialized); + renameSync(temporaryPath, outputPath); } catch (error) { + rmSync(temporaryPath, { force: true }); if (!isAlreadyExistsError(error)) throw error;
renameSync는 기존 파일을 덮어쓰므로, 충돌 검사를 유지하려면 rename 전에 기존 파일 내용을 비교하는 순서로 재구성하십시오.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/testing/src/libs/test-evidence-files.ts` around lines 29 - 39, Update the evidence write flow around writeFileSync and the existing EEXIST handling to write serialized content to a unique temporary file first, then compare any existing outputPath before publishing with renameSync. Preserve the collision error for differing completed content, remove the temporary file on failed or redundant publication, and ensure rename does not overwrite an existing file.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/testing/package.json`:
- Around line 29-37: Update the package exports for ./playwright-reporter and
./vitest-reporter so the types condition precedes import and require, mapping
each module condition to its corresponding declaration file format, d.mts for
ESM and d.cts for CJS. Update the tsup configuration at the referenced build
setup to emit both declaration files and ensure the exports point to them.
In `@packages/testing/schemas/test-evidence-bundle-v1.schema.json`:
- Line 25: Update the $id values in
packages/testing/schemas/test-evidence-bundle-v1.schema.json:25-25 and
packages/testing/schemas/test-evidence-v1.schema.json:3-3 so each ends with its
exact schema filename, preserving the relative $ref resolution. In
packages/testing/src/tests/TestEvidence.spec.ts:55-71, add assertions verifying
both schemas’ $id final path segments match their respective filenames.
In `@packages/testing/schemas/test-evidence-v1.schema.json`:
- Line 42: Update the attempts schema and its attempt definition to enforce
unique attempt numbers where supported, and add a description documenting that
values must be consecutive starting at 1. Keep the existing minimum constraint
and runtime validation behavior aligned with this documented contract.
In `@packages/testing/src/libs/test-evidence-reporters.ts`:
- Around line 214-226: Update the default attempts mapping in the test evidence
reporter to sort results by PlaywrightTestResult.retry before mapping them, and
derive each attempt number from retry (retry + 1) instead of the array index.
Keep the existing attachment, duration, and outcome handling unchanged, and
leave the custom this.options.attempts callback behavior intact.
- Around line 195-211: In the diagnostics construction around the test result
handling, define the failed-result predicate once and reuse it for both error
collection and the failure flag passed to failureDiagnostics. Update the filter
and some calls to reference that shared predicate so diagnostics selection and
failure detection cannot diverge.
In `@packages/testing/src/libs/test-evidence.mts`:
- Around line 397-416: Update the missingArtifacts validation around
bundle.missingArtifacts.forEach to verify each artifact.recordId exists in the
bundle’s records ID set before accepting it. Reject unknown recordId values with
TestEvidenceContractError, while preserving the existing non-empty and required
checks.
- Around line 182-189: Update createTestEvidenceRecord to validate that
input.intent, input.replay, and input.attempts is an array before dereferencing
or spreading them, and raise the established Problem type with stable diagnostic
codes for invalid shapes. Perform these guards before sorting attempts, while
preserving the existing validation flow for valid input.
- Around line 433-441: Extract the shared status derivation used by the
validation block and createTestEvidenceBundle into a single helper, analogous to
summarizeRecords. Update both callers to use this helper while preserving the
existing failed-versus-passed rules based on failed, flaky, and missing-artifact
counts.
In `@packages/testing/src/tests/TestEvidence.spec.ts`:
- Around line 244-256: In the test case “redacts structured tokens and private
keys from ordinary string fields,” add a concise comment next to the
AKIAIOSFODNN7EXAMPLE fixture stating that it is an intentional AWS documentation
example, not a real credential, so static scanners and reviewers recognize the
expected test value.
- Around line 55-71: 보이는 스키마 동기화 테스트에 각 스키마의 $id를 검증하는 단정을 추가하십시오.
test-evidence-v1 스키마와 test-evidence-bundle-v1 스키마의 $id 마지막 경로 세그먼트가 각각 해당 JSON
파일명과 일치하는지 확인하고, 기존 schemaVersion 및 $ref 검증은 유지하십시오.
In `@scripts/release-spine-evidence.mts`:
- Around line 1083-1103: Fallback execution must clear reused-evidence metadata
when required artifacts are missing. In the fallback path following the failed
collectReusedArtifactReferences check, update the result derived from
report.checks[index] so reusedEvidence is explicitly undefined or omitted before
the rerun update at updateCheck, while preserving reuse metadata for the
successful reuse path. Add a test covering missing required artifacts and verify
the final report does not mark the rerun as reused.
In `@scripts/test-evidence-bundle.mts`:
- Around line 212-236: Update parseArguments to replace its generic TypeError
throws for missing --input, missing --output, unknown arguments, and missing
required input paths with the established Problem subclass and Problem code
pattern used by release-spine-evidence. Preserve the existing validation
conditions and messages while ensuring each CLI parsing failure exposes the
appropriate Problem code.
- Around line 91-138: Update the evidence flow across
EvidenceCommand/EvidenceCheckResult, verification-manifest check definitions,
and ReleaseEvidenceCheck to carry each check’s actual fidelity. In the record
construction within test-evidence-bundle, replace the hardcoded fidelity values
with the propagated or category/check-ID mapping, preserving isolated fidelity
for typecheck/public-api and resource-backed fidelity for CORE_COVERAGE or
test:real checks.
In `@scripts/tests/public-api-surface.spec.ts`:
- Around line 113-136: Extend the test covering getCodeEntrypoint and
resolveLocalModule to also verify a .cjs re-export resolves to the corresponding
.cts source declaration. Add a separate .cjs/.cts case or parameterize the
existing .mjs/.mts case, preserving the current assertions for declarationKind,
name, and source.
---
Outside diff comments:
In `@scripts/verification-manifest.mts`:
- Around line 543-558: Update the test command in the “test” manifest entry to
avoid unconditionally passing Turbo’s “--force” flag, allowing cached test
results to be reused; preserve the summarized-test behavior and ensure cached
executions still restore the required evidence files when supported by the
existing verification flow.
---
Duplicate comments:
In `@packages/testing/src/libs/test-evidence-files.ts`:
- Around line 29-39: Update the evidence write flow around writeFileSync and the
existing EEXIST handling to write serialized content to a unique temporary file
first, then compare any existing outputPath before publishing with renameSync.
Preserve the collision error for differing completed content, remove the
temporary file on failed or redundant publication, and ensure rename does not
overwrite an existing file.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 045c54c2-6bde-4830-bcd4-65af7cc2a1f9
📒 Files selected for processing (76)
.changeset/unified-test-evidence.md.github/workflows/ci.ymlpackage.jsonpackages/docs/src/content/docs/api/problems-core/src/classes/Problem.mdpackages/docs/src/content/docs/api/problems-core/src/variables/CROCO_PROBLEM_CODE_REGISTRY.mdpackages/docs/src/content/docs/api/testing/src/classes/CrocoPlaywrightEvidenceReporter.mdpackages/docs/src/content/docs/api/testing/src/classes/CrocoVitestEvidenceReporter.mdpackages/docs/src/content/docs/api/testing/src/classes/TestEvidenceContractError.mdpackages/docs/src/content/docs/api/testing/src/classes/TestEvidenceFidelityError.mdpackages/docs/src/content/docs/api/testing/src/functions/assertNoTestEvidenceSecrets.mdpackages/docs/src/content/docs/api/testing/src/functions/assertTestEvidenceBundle.mdpackages/docs/src/content/docs/api/testing/src/functions/assertTestEvidenceFidelity.mdpackages/docs/src/content/docs/api/testing/src/functions/assertTestEvidenceRecord.mdpackages/docs/src/content/docs/api/testing/src/functions/classifyTestEvidenceOutcome.mdpackages/docs/src/content/docs/api/testing/src/functions/createTestEvidenceBundle.mdpackages/docs/src/content/docs/api/testing/src/functions/createTestEvidenceFileWriter.mdpackages/docs/src/content/docs/api/testing/src/functions/createTestEvidenceRecord.mdpackages/docs/src/content/docs/api/testing/src/functions/createTestKernelEvidenceRecord.mdpackages/docs/src/content/docs/api/testing/src/functions/redactTestEvidence.mdpackages/docs/src/content/docs/api/testing/src/functions/renderTestEvidenceMarkdown.mdpackages/docs/src/content/docs/api/testing/src/functions/serializeTestEvidence.mdpackages/docs/src/content/docs/api/testing/src/functions/testEvidenceFidelityFromKernel.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/PlaywrightTestCase.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/PlaywrightTestResult.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceArtifactProbe.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceAttachment.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceAttempt.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceAttemptOutcome.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceBundle.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceDiagnostic.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceFidelity.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceFidelityRequirement.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceFileWriterOptions.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceIntent.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceJsonValue.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceMissingArtifact.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceObservation.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceOutcome.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceRecord.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceRecordInput.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceReplay.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceReporterContext.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceReporterOptions.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceResourceStatus.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceRunner.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestEvidenceTiming.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/TestKernelEvidenceRecordInput.mdpackages/docs/src/content/docs/api/testing/src/type-aliases/VitestTask.mdpackages/docs/src/content/docs/api/testing/src/variables/TEST_EVIDENCE_SCHEMA_VERSION.mdpackages/docs/src/content/docs/api/webhooks-core/src/classes/UnknownWebhookEventProblem.mdpackages/testing/README.mdpackages/testing/package.jsonpackages/testing/schemas/test-evidence-bundle-v1.schema.jsonpackages/testing/schemas/test-evidence-v1.schema.jsonpackages/testing/scripts/copy-schemas.mjspackages/testing/src/index.tspackages/testing/src/libs/test-evidence-files.tspackages/testing/src/libs/test-evidence-reporters.tspackages/testing/src/libs/test-evidence.mtspackages/testing/src/playwright-reporter.tspackages/testing/src/tests/TestEvidence.spec.tspackages/testing/src/vitest-reporter.tspackages/testing/vitest.config.tspublic-api-surface.snapshot.jsonscripts/public-api-surface.mtsscripts/release-spine-evidence.mtsscripts/test-evidence-bundle.mtsscripts/tests/ci-workflow.spec.tsscripts/tests/public-api-surface.spec.tsscripts/tests/release-spine-evidence.spec.tsscripts/tests/test-evidence-bundle.spec.tsscripts/tests/turbo-task-contract.spec.tsscripts/tests/verification-manifest.spec.tsscripts/verification-manifest.mtsscripts/workflow-verification-contract.mtsturbo.json
Member
Author
|
CodeRabbit outside-diff finding follow-up: The unconditional --force is intentional and retained. CROCO_TEST_EVIDENCE_DIR points to a shared CI directory outside each Turbo task output. Adding it as a hashed env input only changes cache keys; it does not recreate native per-run evidence on a cache hit. Declaring the shared global directory as every package test task output would also create overlapping output ownership and unsafe cache restoration. The verification profile therefore forces affected test execution so the evidence describes the current invocation, while the CI fallback remains only for unsupported reporters. This path passed the repository performance budget, full 234/234 pre-push tests, and the independent adversarial review. |
kang-heewon
force-pushed
the
issue-1487-unified-test-evidence
branch
from
August 1, 2026 19:32
17e57b6 to
7cbbc6d
Compare
kang-heewon
force-pushed
the
issue-1487-unified-test-evidence
branch
3 times, most recently
from
August 1, 2026 21:52
b352683 to
8b41e6e
Compare
kang-heewon
force-pushed
the
issue-1487-unified-test-evidence
branch
from
August 1, 2026 22:19
8b41e6e to
474001a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Outcome
@croco/testingnow publishes the versionedcroco.test-evidence/v1TypeScript and JSON-schema contract, deterministic JSON/Markdown bundling, strict runtime validation, fidelity enforcement, redaction, and direct-load Vitest and Playwright reporters. Retry-then-pass remains visible as flaky with every supplied attempt, while declared intent stays separate from runtime observations.Existing verification/provider/failure-drill artifacts can be normalized without losing their source schema. Release profiles may reuse exact-head evidence only when command, profile, provenance, and required artifacts still match. CI forwards the reporter destination, bypasses cached side-effect loss with a dedicated producer when necessary, fails closed on missing native evidence, and publishes one bundle and one summary.
Fixes #1487
Verification
pnpm --filter @croco/testing test— 134/134 tests passed on the current head.pnpm --filter @croco/testing typecheckand earlier package lint/build checks passed.pnpm public-api:check— 115 package API snapshots matched.pnpm changeset-required:check -- --base origin/trunk --head HEAD— publishable changes covered.pnpm check— 24/25 gates passed, 1 not applicable, 0 failed on the current head.Review gates
@croco/testing, external inputs reject unknown fields, and structured tokens/private keys plus configured secret samples are redacted and revalidated at the bundle boundary.Residual risk
Only the
@croco/testingVitest suite is guaranteed as the default native CI producer. Application, real-resource, failure-drill, and Playwright integrations use the common adapters/normalizer and combined validation coverage, but individual downstream suites must opt into their fidelity and observation mappers as they adopt the format.Summary by CodeRabbit
새 기능
문서
CI 개선