You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Author rich-form evals/evals.json for five judgment-bearing skills, one file per skill under plugins/<plugin>/skills/<skill>/evals/. Each ships 6 cases covering trigger/routing, the happy path, at least one refusal/guardrail, and at least one must-not-do anti-pattern, with objectively-verifiable expectations.
discovery/explore
discovery/explore-deep
discovery/research
discovery/research-deep
claude-config-audit/memory-health
Variant skills exercise their distinctive contract rather than re-testing the parent:
explore-deep — fork isolation, read-only boundary (must not edit source), summary+artifact-path return (not the full 7-section report), fallback when CLAUDE_CODE_FORK_SUBAGENT unset, empty-scope handling, sidecar when EXPLORE.md exists.
research-deep — dispatcher behavior: multi-topic → N parallel agents (not one blob on the engine), tier selection, does-not-run-inline, full discipline even on a small task.
Per-skill warrant verdicts (re-checked against live SKILL.md)
Dispatcher decisions (multi-topic split, tier selection, no-inline-deep-pass) are judgment calls
claude-config-audit/memory-health
WARRANTED
Scope boundary, contributor-local persistence, fix-gating, consumer-exemption handling are behavioral
No skill in this batch was pure-reference; no skip verdicts.
Version bumps
Both touched plugins bumped 0.1.0 → 0.2.0 (minor — evals are an additive shipped component; a version bump is the only delivery vehicle). Matches the precedent of a minor bump for adding a new shipped component (implementation plugin → v0.2.0 for a setup skill, #100).
Validation
All 5 files pass check-jsonschema against plugins/skill-quality/reference/evals.schema.json.
All 5 skills pass check-skill.sh (0 errors); no evals-missing WARN, confirming presence is detected. Residual WARNs are pre-existing skill-body items (Gotchas surface / Use when: phrasing), out of scope for this evals-authoring task.
Note on stale issue-body references
The claimed issue body cites two paths that do not exist on main: docs/MIGRATION-PLAYBOOK.md "Evals — warrant policy and consumer-verify recipe" and docs/evals-coverage.md. The warrant guidance actually lives in plugins/skill-quality/skills/skill-quality/SKILL.md (validate-evals action) — used as the warrant authority here. Flagging for tracker hygiene; not a blocker.
Refs melodic-software/medley#1448
Note
Low Risk
Additive eval JSON and minor plugin version bumps only; no changes to skill runtime logic or application code paths.
Overview
Adds rich-formevals/evals.json (six cases each, with prompt, expected_output, and verifiable expectations) for five judgment-bearing skills: discoveryexplore, explore-deep, research, research-deep, and claude-config-auditmemory-health.
The new cases encode each skill’s behavioral contract—happy paths, scope/routing, guardrails (read-only, no auto-fix, no training-recall claims), and variant-specific rules (e.g. fork isolation and sidecar artifacts for explore-deep; dispatcher tiering and parallel topic agents for research-deep; contributor-local reports and fix gating for memory-health).
Plugin versions are bumped as the delivery vehicle for the new shipped eval assets: claude-config-audit0.1.0 → 0.2.0, discovery0.2.0 → 0.3.0.
Reviewed by Cursor Bugbot for commit 9406015. Bugbot is set up for automated code reviews on this repo. Configure here.
…lls)
Rich-form evals/evals.json for discovery/{explore,explore-deep,research,
research-deep} and claude-config-audit/memory-health. Each ships 6 cases
spanning trigger/routing, happy path, a refusal/guardrail, and a must-not-do
anti-pattern, with objectively-verifiable expectations. Variant skills exercise
their distinctive contracts (fork isolation + read-only boundary for
explore-deep; multi-topic dispatch + tier selection for research-deep).
Minor version bump on both touched plugins (0.1.0 -> 0.2.0) — evals are a
shipped component and a version bump is the only delivery vehicle.
All five validated against plugins/skill-quality/reference/evals.schema.json
(check-jsonschema) and pass check-skill.sh.
Refs melodic-software/medley#1448
Validate evals.json structure and schema compliance
Check plugin.json version bumps
Assess eval case quality (coverage, expectations, guardrails)
Post findings
Summary
This PR adds 30 eval cases across five skills. The structure is schema-compliant, version bumps are appropriate, and the variant skills test their distinctive behavioral contracts rather than duplicating the parent. The expectations are objectively verifiable behavioral tests — not subjective quality checks — which is the right form. I verified all cases against their live SKILL.md contracts.
Two medium issues (missing expectations that leave SKILL.md contracts untested) and two minor issues (a weakened hedge, a name collision) follow.
Medium — missing expectation coverage
M1: explore case 2 (blindspot-builds-user-knowledge) — intake question not tested
The explore SKILL.md defines a mandatory first step for blindspot mode:
Intake — ask the user's starting point first (one question). Blindspot output calibrates to that disclosure.
This is a meaningful behavioral contract (the scan content depends on the disclosed starting point), but no expectation in the case verifies that the skill asks this intake question before scanning. The current four expectations test the card format, the improved-prompt assembly, and the no-EXPLORE.md default — the intake gate is unguarded.
Suggested additional expectation:
"Output asks the user one intake question about their starting point before or at the start of the scan"
M2: explore case 6 (asks-before-deleted-file-archaeology) — positive continuation behavior not in expectations
expected_output reads: "…asks the user before investigating the removed file via git archaeology, rather than silently reconstructing it. Continues the rest of the config-dimension exploration normally." The second clause ("continues the rest of the config-dimension exploration normally") is an important contract — the skill must not abort the whole exploration, only gate the deleted-file sub-task. But the expectations list only verifies the ask behavior; the continuation is absent.
Only two expectations are present (vs. three in every other case across all five files), which is how the gap surfaced.
Suggested third expectation:
"Output continues exploring the rest of the config dimension normally despite gating the deleted-file archaeology"
m1: explore case 1 expectation 4 — hedge weakens the artifact-write contract
"The run persists an EXPLORE.md artifact (or names the artifact path it would write) for handoff"
The explore SKILL.md says write the file, unconditionally (no fallback to "name the path"). The hedge suggests an acceptable alternative that the skill spec doesn't allow. An evaluator running this case could pass an output that only declares a path without writing — which is a behavioral regression.
Suggested: drop the parenthetical and assert Write was called.
m2: duplicate name across research and research-deep
research case 5 and research-deep case 6 both use "name": "researches-does-not-decide-or-code". The schema scopes name per file so there's no schema violation. However, eval tooling or reporters that surface case names without file context will collide on this label. The research-deep case is distinct enough to carry a unique name — something like dispatcher-researches-does-not-decide would disambiguate while keeping the behavioral signal.
✓ — fork isolation, read-only boundary, return-contract, fallback, empty-scope, sidecar all tested and aligned with SKILL.md
research-deep dispatcher behavior
✓ — multi-topic → N parallel agents, tier selection, no-inline-deep-pass all correct
research falsification mandate
✓ — case 3 correctly tests that Phase 2 must include an explicit falsification query
memory-health scope routing
✓ — MCP, hooks, agents correctly routed to settings-audit/automation-deep-dive
memory-health contributor-local persistence
✓ — aligned with "never in the consuming repo" contract in SKILL.md
memory-health fix-gating
✓ — aligned with "requires prior audit, asks for approval" in SKILL.md
memory-health REPO check-ID seam
✓ — exemption surfaced under distinct REPO check-ID as the extension seam specifies
Version bumps (0.1.0 → 0.2.0)
✓ — minor bump is appropriate for additive shipped component
Security
✓ — pure JSON test definitions; no code execution, no secrets, no external URLs
The two medium items are worth fixing before merge — both leave documented SKILL.md contracts with no verifiable expectation, which is the main thing evals exist to catch. The minor items are optional-but-clean. Overall a well-authored batch with tight alignment to the live skill specs.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Did not approve because Cursor Bugbot completed with skip status (usage limit reached), so the required automated review did not complete. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Validate evals.json structure and case quality against live SKILL.md contracts
Check version bumps
Post findings
Summary
30 eval cases across five skills. Schema is valid (all files pass the evals.schema.json shape), version bumps are appropriate, and the variant skills (explore-deep, research-deep) correctly exercise their distinctive behavioral contracts rather than duplicating the parent. I verified every expectation against the live SKILL.md for each skill.
Two medium issues (SKILL.md-documented contracts without a covering expectation) and three minor issues (two hedge wordings that weaken write-contract assertions, one name collision) follow.
Medium — undercovered behavioral contracts
M1 — explore case 2: blindspot intake question not tested
SKILL.md line 124 mandates a first step before scanning:
Intake — ask the user's starting point first (one question). Blindspot output calibrates to that disclosure.
The expected_output describes this ("asks one intake question about the user's starting point") but none of the four expectations assert it. An evaluator can pass output that skips the intake gate entirely and goes straight to blindspot cards.
Suggested addition:
"Output asks the user one intake question about their starting point before beginning the scan"
M2 — explore case 6: continuation behavior described but not tested
expected_output closes with: "Continues the rest of the config-dimension exploration normally." That clause is a second behavioral contract (the skill must not abort the entire exploration, only gate the deleted-file sub-task). Only two expectations are present; the second clause has no covering assertion.
Suggested addition:
"Output continues exploring the rest of the config dimension normally despite gating the deleted-file archaeology"
m1 — explore case 1, expectation 4: hedge weakens the write contract
"The run persists an EXPLORE.md artifact (or names the artifact path it would write) for handoff"
explore SKILL.md line 156 says write the file (write the exploration output to ...), with no fallback. The parenthetical allows an output that only declares a path without writing — a regression the contract doesn't permit.
Suggest dropping the parenthetical and asserting the file was written.
m2 — research case 1, expectation 4: same hedge pattern
"Output reports an outcome-gate result and persists (or names) a RESEARCH.md artifact"
research SKILL.md line 167 likewise says write the artifact. Same issue as m1 — the hedge allows naming-without-writing when the contract requires the write.
m3 — duplicate name across research case 5 and research-deep case 6
Both carry "name": "researches-does-not-decide-or-code". No schema violation (names are scoped per file), but eval reporters that surface case names without file context will collide. The research-deep case is distinct (it's about dispatcher behavior) — a name like dispatcher-researches-does-not-decide would disambiguate without losing the behavioral signal.
✓ — case 4 tests all three: no silent edit, prior audit required, approval asked
memory-health REPO check-ID extension seam
✓ — case 5 aligned with consumer-convention extension seam in SKILL.md
memory-health report-without-audit
✓ — case 6 aligned with report mode ("if missing, inform user… suggest running audit")
Version bumps (0.1.0 → 0.2.0)
✓ — minor bump correct for additive shipped component
Security
✓ — pure JSON test fixtures; no code execution, no secrets, no external URLs
The two medium items (M1, M2) leave SKILL.md-specified behavioral gates uncovered and are worth fixing before merge. The three minor items (m1, m2, m3) are lower priority but clean up contract precision and tooling ergonomics. The overall batch is well-structured and tightly aligned with the live skill specs.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Did not approve because Cursor Bugbot completed with skip status (usage limit reached), so the required automated review did not complete. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Validate evals.json structure against schema and live SKILL.md contracts
Check version bumps
Post findings
Summary
30 eval cases across five skills. Schema is valid (skill_name, evals, id, prompt, kebab-case name, expected_output, files, expectations all structurally correct). Version bumps from 0.1.0 → 0.2.0 are appropriate for an additive shipped component. I verified every expectation against the current SKILL.md for each skill on this branch.
Two medium issues leave documented SKILL.md behavioral contracts with no covering expectation. Three minor issues weaken assertion precision and create a name collision risk.
Medium — uncovered SKILL.md contracts
M1 — explore case 2: blindspot intake question not tested
explore/SKILL.md line 124 defines a mandatory first step:
Intake — ask the user's starting point first (one question). Blindspot output calibrates to that disclosure.
The expected_output for case 2 correctly describes this behavior ("asks one intake question about the user's starting point"), but none of the four expectations assert it. An evaluator can pass output that jumps straight to blindspot cards without the intake gate.
Suggested addition:
"Output asks the user one intake question about their starting point before beginning the blindspot scan"
M2 — explore case 6: continuation behavior in expected_output has no covering expectation
expected_output closes with: "Continues the rest of the config-dimension exploration normally." This is a second behavioral contract (the skill must not abort the whole exploration, only gate the deleted-file sub-task). Only two expectations are present — the continuation clause is entirely unguarded.
Suggested addition:
"Output continues exploring the rest of the config dimension normally despite gating the deleted-file archaeology"
m1 — explore case 1 expectation 4: hedge weakens the write contract
"The run persists an EXPLORE.md artifact (or names the artifact path it would write) for handoff"
explore/SKILL.md line 156 says write the file unconditionally: "write the exploration output to ${user_config.notes_dir}/<topic-slug>/EXPLORE.md" — no fallback. The parenthetical allows an output that only declares a path without writing, which is a behavioral regression that should fail this eval.
m2 — research case 1 expectation 4: same hedge pattern
"Output reports an outcome-gate result and persists (or names) a RESEARCH.md artifact"
research/SKILL.md line 167 likewise mandates writing: "Write the research output to ${user_config.notes_dir}/<topic-slug>/RESEARCH.md". Same issue — the hedge lets a path-declaration pass where a write is required.
m3 — duplicate name across research case 5 and research-deep case 6
Both carry "name": "researches-does-not-decide-or-code". No schema violation (names are per-file scoped), but eval reporters surfacing case names without file context will collide. The research-deep case is semantically distinct — it's testing dispatcher behavior — so a name like dispatcher-researches-does-not-decide would disambiguate without losing the behavioral signal.
✓ — case 1 correctly matches the "N ≥ 2 separable topics" check
research-deep single heavy topic → engine dispatch
✓ — case 2 (3-ORM comparison = one heavy topic, not multi-topic)
research-deep Tier 2 fallback (general-purpose, not read-only Explore)
✓ — case 3 aligned; SKILL.md explicitly requires general-purpose for Phase 3 tool access
research-deep dispatcher-not-inline contract
✓ — case 5 correctly tests "main session stays a dispatch/summary"
research-deep small task still runs full discipline inline
✓ — case 4 covers both routing (Tier 3) and depth invariant
memory-health scope boundary, routes MCP/hooks/agents out
✓ — case 2 aligned with SKILL.md scope table
memory-health contributor-local persistence
✓ — case 3 aligned with "never in the consuming repo"
memory-health fix-gating (prior audit + approval)
✓ — case 4 tests all three gates: no silent edit, prior audit required, approval asked
memory-health REPO check-ID extension seam
✓ — case 5 aligned with consumer-convention seam in SKILL.md
memory-health report-without-audit (no fabrication)
✓ — case 6 aligned with report-mode contract ("if missing, inform user… suggest running audit")
Version bumps (0.1.0 → 0.2.0)
✓ — minor bump appropriate for additive shipped component
Security
✓ — pure JSON test fixtures; no code execution, no secrets, no external URLs
M1 and M2 are worth fixing before merge — both leave SKILL.md-specified behavioral gates uncovered, which is the main thing evals exist to catch. The minor items (m1, m2, m3) are lower priority but tighten contract precision and reduce tooling friction.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Did not approve because Cursor Bugbot completed with skip status (usage limit reached), so the required automated review did not complete. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Check git log — this trigger is a merge-of-main commit; eval files unchanged since original commit c44a094
Validate evals against SKILL.md contracts
Post findings
Context
This PR synchronize is a merge-of-main commit (66cbd3f), not a fix push. The five eval files are unchanged from the original commit. The issues identified in prior reviews remain open — I've re-verified them below against the live SKILL.md contracts.
Medium — undercovered SKILL.md contracts (unchanged from prior review)
M1 — explore case 2: blindspot intake question not covered by any expectation
Intake — ask the user's starting point first (one question). Blindspot output calibrates to that disclosure.
The expected_output correctly describes this behavior, but none of the four expectations assert it. An evaluator can pass output that skips the intake gate and goes straight to blindspot cards.
M2 — explore case 6: continuation behavior in expected_output has no covering expectation
expected_output closes with: "Continues the rest of the config-dimension exploration normally." This is a second behavioral contract — the skill must not abort the whole exploration, only gate the deleted-file sub-task. Only two expectations exist; the continuation is entirely unguarded.
"The run persists an EXPLORE.md artifact (or names the artifact path it would write) for handoff"
explore/SKILL.md line 156 says write unconditionally — no fallback. The parenthetical allows output that only declares a path without writing, which should fail this eval. Suggest dropping (or names the artifact path it would write).
m2 — research case 1 expectation 4: same hedge pattern
m3 — duplicate name across research case 5 and research-deep case 6
Both carry "name": "researches-does-not-decide-or-code". No schema violation (names are per-file scoped), but eval reporters surfacing case names without file context will collide. The research-deep case tests dispatcher behavior and could be named dispatcher-researches-does-not-decide instead.
✓ — case 1 correctly matches the "N ≥ 2 separable topics" check
research-deep single heavy topic → engine dispatch
✓ — case 2 (3-ORM comparison = one heavy topic, not multi-topic)
research-deep Tier 2 fallback (general-purpose, not read-only Explore)
✓ — case 3 aligned; SKILL.md explicitly requires general-purpose for Phase 3 tool access
research-deep dispatcher-not-inline contract
✓ — case 5 correctly tests "main session stays a dispatch/summary"
memory-health scope boundary, routes MCP/hooks/agents out
✓ — case 2 aligned with SKILL.md scope table
memory-health contributor-local persistence
✓ — case 3 aligned with "never in the consuming repo"
memory-health fix-gating (prior audit + approval)
✓ — case 4 tests all three gates
memory-health REPO check-ID extension seam
✓ — case 5 aligned with consumer-convention seam in SKILL.md
memory-health report-without-audit (no fabrication)
✓ — case 6 aligned with report-mode contract
Version bumps (0.1.0 → 0.2.0)
✓ — minor bump appropriate for additive shipped component
Security
✓ — pure JSON test fixtures; no code execution, no secrets, no external URLs
M1 and M2 are the priority — both leave documented SKILL.md behavioral contracts with no verifiable expectation, which is what evals exist to catch. M1 (intake gate) and M2 (continuation after gating) are one expectation each. The minor items (m1, m2, m3) are lower-priority but tighten contract precision and avoid tooling collisions.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Did not approve because Cursor Bugbot completed with skip status (usage limit reached), so the required automated review did not complete. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
The latest push (b23a899) merges origin/main into the branch — it brings in a new discovery/skills/setup/SKILL.md and a README update only. The five eval files are unchanged from the original commit c44a094. All issues from prior reviews remain open.
Medium — undercovered SKILL.md contracts (still open)
M1 — explore case 2: blindspot intake question not tested
Intake — ask the user's starting point first (one question). Blindspot output calibrates to that disclosure.
The expected_output correctly describes this: "asks one intake question about the user's starting point" — but none of the four expectations assert it. An evaluator can pass output that skips the intake gate entirely and jumps straight to blindspot cards.
M2 — explore case 6: continuation behavior described in expected_output has no covering expectation
The expected_output states: "Continues the rest of the config-dimension exploration normally." That clause is a second behavioral contract: the skill must not abort the entire exploration, only gate the deleted-file sub-task. Only two expectations are present; the continuation is entirely unguarded — an output that gates correctly but aborts all remaining exploration would still pass.
"The run persists an EXPLORE.md artifact (or names the artifact path it would write) for handoff"
explore/SKILL.md line 156 says write unconditionally. The parenthetical allows output that only declares a path without writing — which should fail this eval.
Suggested fix: drop (or names the artifact path it would write).
m2 — research case 1 expectation 4: same hedge pattern
"Output reports an outcome-gate result and persists (or names) a RESEARCH.md artifact"
research/SKILL.md line 167 likewise mandates writing: "Write the research output to ${user_config.notes_dir}/<topic-slug>/RESEARCH.md". Same issue as m1.
m3 — duplicate name across research case 5 and research-deep case 6
Both carry "name": "researches-does-not-decide-or-code". No schema violation (names are per-file scoped), but eval reporters surfacing case names without file context will collide. The research-deep case tests dispatcher behavior and could be named dispatcher-researches-does-not-decide to disambiguate.
✓ — case 1 correctly matches the "N ≥ 2 separable topics" check
research-deep single heavy topic → engine dispatch
✓ — case 2 (3-ORM comparison = one heavy topic, not multi-topic)
research-deep Tier 2 fallback (general-purpose, not read-only Explore)
✓ — case 3 aligned; SKILL.md explicitly requires general-purpose for Phase 3 tool access
research-deep dispatcher-not-inline contract
✓ — case 5 correctly tests "main session stays a dispatch/summary"
memory-health scope boundary, routes MCP/hooks/agents out
✓ — case 2 aligned with SKILL.md scope table
memory-health contributor-local persistence
✓ — case 3 aligned with "never in the consuming repo"
memory-health fix-gating (prior audit + approval)
✓ — case 4 tests all three gates
memory-health REPO check-ID extension seam
✓ — case 5 aligned with consumer-convention seam in SKILL.md
memory-health report-without-audit (no fabrication)
✓ — case 6 aligned with report-mode contract
Version bumps (0.1.0 → 0.2.0)
✓ — minor bump appropriate for additive shipped component
Security
✓ — pure JSON test fixtures; no code execution, no secrets, no external URLs
M1 and M2 are the priority — both leave SKILL.md-specified behavioral gates with no verifiable expectation, which is what evals exist to catch. Each is a one-line addition. The minor items (m1, m2, m3) are lower-priority but tighten contract precision and avoid tooling collisions.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Did not approve because Cursor Bugbot completed with skip status (usage limit reached), so the required automated review did not complete. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Did not approve because Cursor Bugbot completed with skip status (usage limit reached), so the required automated review did not complete. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Base main advanced to discovery@0.2.0 (PR #111) while this PR was open, so
the eval files added here no longer shipped with a version bump relative to
base. Bump to 0.3.0 so consumers receive the new eval assets on marketplace
update. Addresses Codex review finding.
Fixed in edd90ce — bumped discovery to 0.3.0. Correct catch: PR #111 merged discovery@0.2.0 to main while this PR was open, so after pulling main the eval files here no longer shipped with a bump relative to base. 0.3.0 restores delivery. (claude-config-audit@0.2.0 bump was already intact vs its main baseline of 0.1.0.)
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Did not approve because Cursor Bugbot completed with skip status (usage limit reached), so the required automated review did not complete. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Check open issues from prior reviews (M1, M2, m1, m2, m3)
Post findings
Summary
30 eval cases across five skills. Schema is valid, the version bump issue (discovery 0.2.0 → 0.3.0) is correctly resolved in edd90ce — both plugins are now properly bumped relative to their main baselines. I re-verified every expectation against the live SKILL.md for each skill at HEAD.
The five issues from prior reviews are all still open — the eval files are unchanged from the original commit. Two medium issues leave SKILL.md-documented behavioral gates with no covering expectation; three minor issues weaken assertion precision and create a name collision.
Medium — undercovered SKILL.md contracts
M1 — explore case 2: blindspot intake question not tested
Intake — ask the user's starting point first (one question). Blindspot output calibrates to that disclosure.
The expected_output for case 2 correctly describes this behavior ("asks one intake question about the user's starting point"), but none of the four expectations assert it. An evaluator can pass output that skips the intake gate and jumps straight to blindspot cards — exactly the regression this case should catch.
M2 — explore case 6: continuation behavior in expected_output has no covering expectation
The expected_output closes with: "Continues the rest of the config-dimension exploration normally." This is a second behavioral contract — the skill must not abort the whole exploration, only gate the deleted-file sub-task. Only two expectations are present; an output that correctly asks before archaeology but then stops all further exploration would still pass.
"The run persists an EXPLORE.md artifact (or names the artifact path it would write) for handoff"
explore/SKILL.md line 156 says write unconditionally — no fallback. The parenthetical allows output that only declares a path without writing, which is a behavioral regression that should fail this eval. Suggest dropping (or names the artifact path it would write).
m2 — research case 1 expectation 4: same hedge pattern
"Output reports an outcome-gate result and persists (or names) a RESEARCH.md artifact"
research/SKILL.md line 167 likewise mandates writing unconditionally. The hedge (or names) lets a path-declaration pass where a write is required.
m3 — duplicate name across research case 5 and research-deep case 6
Both carry "name": "researches-does-not-decide-or-code". No schema violation (names are per-file scoped), but eval reporters surfacing case names without file context will collide. The research-deep case is semantically distinct — it tests dispatcher behavior. A name like dispatcher-researches-does-not-decide would disambiguate without losing the behavioral signal.
✓ — case 1 correctly matches the "N ≥ 2 separable topics" check
research-deep single heavy topic → engine dispatch
✓ — case 2 (3-ORM comparison = one heavy topic, not multi-topic)
research-deep Tier 2 fallback (general-purpose, not read-only Explore)
✓ — case 3 aligned; SKILL.md explicitly requires general-purpose for Phase 3 tool access
research-deep dispatcher-not-inline contract
✓ — case 5 correctly tests "main session stays a dispatch/summary"
research-deep small task still runs full discipline inline
✓ — case 4 covers both routing (Tier 3) and depth invariant
memory-health scope boundary
✓ — case 2 correctly routes MCP/hooks/agents to settings-audit / automation-deep-dive
memory-health contributor-local persistence
✓ — case 3 aligned with "never in the consuming repo"
memory-health fix-gating (prior audit + approval)
✓ — case 4 tests all three gates: no silent edit, prior audit required, approval asked
memory-health REPO check-ID extension seam
✓ — case 5 aligned with consumer-convention seam in SKILL.md
memory-health report-without-audit (no fabrication)
✓ — case 6 aligned with report-mode contract
Security
✓ — pure JSON test fixtures; no code execution, no secrets, no external URLs
M1 and M2 are the priority fixes — both leave SKILL.md-specified behavioral gates with no verifiable expectation, each requiring one line. The minor items (m1–m3) are lower priority but tighten contract precision and avoid tooling collisions. The overall batch is well-structured; these are targeted gaps in otherwise solid eval authorship.
The reason will be displayed to describe this comment to others. Learn more.
Risk: low. Did not approve because Cursor Bugbot completed with skip status (usage limit reached), so the required automated review did not complete. Human review is needed before merge; no reviewers were assigned because the PR author is the sole human repository contributor.
Sent by Cursor Approval Agent: Pull Request Router and Approver
## What
A new `comment-residue` skill in the **code-tidying** plugin: a
read-only classifier that flags **out-of-context residue** in code
comments and presents Tier 1/2 findings for author-applied deletion.
Mirrors `declutter`'s architecture (`detect.sh` + sourceable shape
library + self-contained `detect.test.sh`), but scans code files and
detects only on the **comment portion** of a line.
**Residue shapes:** `history-narration` ("used to… now…"),
`plan-reference` ("Task 2 replaces the old…", "in this PR"),
`conversational-antecedent` ("per your request") — all Tier 1;
`ticket-pr-residue` ("see PR #45", "JIRA-123") — Tier 2. `TODO(#issue)`
is the sanctioned back-reference and is never flagged.
Bumps code-tidying `0.2.0 → 0.3.0`; documents the skill in the manifest
+ README catalog.
## Why
Agents leak the producing conversation/plan/history into comments — text
that only makes sense inside the chat thread that wrote it. No existing
skill targets this: `declutter` is markdown-only, `compress` excludes
code comments, and `tidy`'s Beck-#15 handles generic restatement, not
the out-of-context residue class. This is the clean-up-afterward
companion to the source-side comment rule (a separate dotfiles-repo
change).
## Design notes
- **Comment-scoped detection** — residue-shaped words in identifiers or
string literals (`previously = load_cache()`, `# encode as UTF-8 /
SHA-256`) are **not** flagged; only the comment text is classified. This
keeps false positives low; anything uncertain lands in Tier 2 (review),
and the skill is read-only so the author confirms every deletion.
- **Placement** — filed under `code-tidying` (subject = code) per the
plan. It's a read-only *classifier* like `declutter` (which lives in
`docs-hygiene`), so a mode-based argument for `docs-hygiene` exists;
flagging for reviewer preference.
## Security review (plugin-acceptance)
Touches **only surface 1** — a read-only, advisory `detect.sh` (spawns
git/find/awk, never mutates, no network, no MCP server, no
`userConfig`/secrets, reads only files under the invoking repo via
`${CLAUDE_PLUGIN_ROOT}` / `git rev-parse`). No surface 2/5/6 trust
decision required.
## Test plan
- [x] `detect.test.sh` — 19/19 (shapes, tiers, opt-out, markdown-skip,
dir + paths-file, and the false-positive guards: code identifier,
`UTF-8`/`SHA-256`, sanctioned `TODO(#123)`)
- [x] shellcheck clean
- [x] shfmt clean (editorconfig-aware)
- [x] markdownlint-cli2 clean (repo config)
- [x] `claude plugin validate --strict .` passed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Read-only local bash/git scanning with no mutations, secrets, or
network; CI change is a targeted hygiene exclude for test/detector
content.
>
> **Overview**
> Adds **`/code-tidying:comment-residue`**, a read-only skill that
audits **code** comments for four residue shapes (history narration,
plan/session refs, conversational antecedents, ticket/PR back-refs) and
reports **Tier 1/2** findings with treatment guidance—authors apply
deletions; no edits from the skill.
>
> Implementation follows the `declutter`-style split: **`detect.sh`**
(targets, git default, deterministic output),
**`lib/comment-shapes.sh`** (comment-only extraction with string-aware
leaders, shape regexes, sanctioned `TODO(#issue)`,
`comment-residue-ignore`), plus **`detect.test.sh`**, eval fixtures, and
**`evals.json`**. **code-tidying** bumps to **0.3.0**; README and
manifest copy describe comment hygiene alongside `tidy` /
`batch-simplify`.
>
> CI **comment-hygiene** now **excludes** `comment-residue` **scripts**
and **evals** so intentional TODO/PR-shaped test corpus and detector
patterns do not trip the repo ban policy.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
43f7bfa. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…npm-minor-patch group across 1 directory (#221)
Bumps the npm-minor-patch group with 1 update in the / directory:
[@anthropic-ai/claude-code](https://github.com/anthropics/claude-code).
Updates `@anthropic-ai/claude-code` from 2.1.210 to 2.1.212
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/anthropics/claude-code/releases">@anthropic-ai/claude-code's
releases</a>.</em></p>
<blockquote>
<h2>v2.1.212</h2>
<h2>What's changed</h2>
<ul>
<li><code>/fork</code> now copies your conversation into a new
background session (its own row in <code>claude agents</code>) while you
keep working; the in-session subagent it used to launch is now
<code>/subtask</code></li>
<li>Added <code>claude auto-mode reset</code> to restore the default
auto-mode configuration, with a confirmation prompt (pass
<code>--yes</code> to skip)</li>
<li>Added a session-wide limit on WebSearch tool calls (default 200,
tunable via <code>CLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION</code>) to
stop runaway search loops</li>
<li>Added a per-session cap on subagent spawns (default 200, override
with <code>CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION</code>) to stop runaway
delegation loops; <code>/clear</code> resets the budget</li>
<li>MCP tool calls running longer than 2 minutes now move to the
background automatically so the session stays usable; configure the
threshold or disable with
<code>CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS</code></li>
<li>Typing <code>/resume</code> in the agent view now opens a picker of
past sessions — including sessions deleted from the list — and resumes
your pick as a background session</li>
<li>Fixed plan mode auto-running file-modifying Bash commands (e.g.
<code>touch</code>, <code>rm</code>) without a permission prompt or SDK
<code>canUseTool</code> callback</li>
<li>Fixed worktree creation following a repository-committed symlink at
<code>.claude/worktrees</code>, which could create files outside the
repository</li>
<li>Fixed a <code>continue:false</code> hook's halt being dropped when
the tool fails or completes mid-stream, and hook infrastructure errors
being misreported as user rejections</li>
<li>Fixed SIGTERM during a running Bash tool orphaning the command's
process tree in print/SDK mode; the CLI now aborts the turn, kills the
tree, and exits 143</li>
<li>Fixed <code>/background</code> and <code>claude --bg</code> failing
with "EUNKNOWN: unknown error, uv_spawn" on Windows when Group
Policy blocks PowerShell 5.1; the daemon now prefers PowerShell 7</li>
<li>Fixed shell mode (<code>!</code>) not executing commands containing
file paths while the path autocomplete popup was open</li>
<li>Fixed auto-mode denial notifications rendering broken characters
when a long denial reason was truncated mid-emoji</li>
<li>Fixed Ctrl+J not inserting a newline in the agent view dispatch
input on terminals with extended key reporting, and surfaced the newline
shortcut in the <code>?</code> help overlay</li>
<li>Fixed <code>/ultrareview</code> rejecting PR references like
<code>[#123](https://github.com/anthropics/claude-code/issues/123)</code>,
<code>PR 123</code>, and pasted PR URLs; error hints now name the
command you actually typed</li>
<li>Fixed <code>/ultrareview <branch></code> not fetching the
branch from origin when it exists remotely; it now suggests the closest
branch name on typos</li>
<li>Fixed <code>/ultrareview</code> skipping the billing confirmation in
a new conversation after <code>/clear</code></li>
<li>Fixed <code>/ultrareview</code>'s "not a git repository"
error on Claude Desktop now suggesting the project's repository folder
instead of terminal commands</li>
<li>Fixed hosted (host-managed) sessions failing at startup when
repository settings configured mTLS certs, extra CA bundles, or OAuth
scopes; these transport settings are now ignored with a warning</li>
<li>Fixed a spurious "File has not been read yet" error when
editing a file that had been read with offset/limit before resuming a
session</li>
<li>Fixed <code>ExitWorktree</code> failing with "no active
EnterWorktree session" after resuming a session with
<code>--continue</code>/<code>--resume</code> in print/SDK mode</li>
<li>Fixed the workflow agent grid staying empty for Remote Control
clients that join a session mid-run</li>
<li>Fixed streaming-mode control requests being marked complete before
their handler finished, which could lose the request on session
restart</li>
<li>Fixed background sessions created with <code>/fork</code> losing
their live-parent protection after a state write failure</li>
<li>Fixed reopening a stopped background session from the agent view
failing silently — it now resumes the session, or shows why it can't and
lets you force a restart</li>
<li>Fixed agent teams: a stopping teammate could send the leader
duplicate idle notifications when team initialization re-ran within a
session</li>
<li>Fixed the plan-approval dialog footer splitting "ctrl+g to edit
in <!-- raw HTML omitted -->" apart when the file path is long</li>
<li>Fixed the welcome banner keeping its old panel widths after a
combined width+height terminal resize in fullscreen mode</li>
<li>Fixed diff previews losing their line numbers and +/- markers in
narrow layouts</li>
<li>Fixed @-mentions attaching nothing after a partial file read, plugin
uninstall targeting the wrong marketplace, and false "Command timed
out" on exit code 143</li>
<li>Fixed OpenTelemetry HTTP exports being rejected with 411/400 by
Azure Monitor and other endpoints that don't accept chunked transfer
encoding</li>
<li>Fixed OTLP event log records missing
<code>trace_id</code>/<code>span_id</code> when <code>TRACEPARENT</code>
is set in SDK/headless mode</li>
<li>Fixed conversations with many images incorrectly failing with
"Request too large" errors, and improved the error message to
explain the actual cause</li>
<li>Fixed web search and web fetch returning "API Error" text
as search results or page content when the API was overloaded</li>
<li>Improved web search and web fetch reliability by retrying 529 errors
and rate-limited requests with bounded backoff</li>
<li>Improved prompt caching: the mid-conversation system block now works
behind LLM gateways and custom base URLs (Bedrock, Vertex, 1P)</li>
<li>Improved background agent attach: cold-attaching now instantly shows
the formatted transcript while the session boots, instead of a blank
wait</li>
<li>Reduced token usage in inter-agent messaging:
<code>SendMessage</code> bodies are no longer duplicated into replayed
history and tool results</li>
<li>Changed <code>/fork</code> to name the copy after your prompt when
the session has no title, so the row is recognizable in the agent
view</li>
<li>Changed bare <code>/btw</code> to reopen the side-question panel on
your most recent exchange so you can browse earlier answers</li>
<li>Changed the <code>←</code> footer hint to pulse <code>N done</code>
for a moment when a background agent finishes while nothing needs your
input</li>
<li>Deprecated the Task tool's <code>mode</code> parameter (now
ignored); subagents inherit the parent session's permission mode by
default</li>
<li>Changed Enterprise <code>forceLoginMethod</code> to be enforced for
VS Code extension, SDK, <code>setup-token</code>, and
<code>install-github-app</code> logins, not just the terminal</li>
<li>Changed session transcripts to record the reasoning effort level on
each assistant message</li>
<li>Changed headless/SDK sessions to apply a <code>set_model</code>
control request mid-turn; the next model round-trip uses the new model
instead of waiting for the next turn</li>
<li>Changed agent view / <code>claude agents --json</code>: sessions
waiting on a sandbox, MCP-input, or managed-settings prompt now show as
"Needs input" instead of "Working"</li>
<li>Updated the auth status panel title from "Cloud
authentication" to "Authentication"</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md">@anthropic-ai/claude-code's
changelog</a>.</em></p>
<blockquote>
<h2>2.1.212</h2>
<ul>
<li><code>/fork</code> now copies your conversation into a new
background session (its own row in <code>claude agents</code>) while you
keep working; the in-session subagent it used to launch is now
<code>/subtask</code></li>
<li>Added <code>claude auto-mode reset</code> to restore the default
auto-mode configuration, with a confirmation prompt (pass
<code>--yes</code> to skip)</li>
<li>Added a session-wide limit on WebSearch tool calls (default 200,
tunable via <code>CLAUDE_CODE_MAX_WEB_SEARCHES_PER_SESSION</code>) to
stop runaway search loops</li>
<li>Added a per-session cap on subagent spawns (default 200, override
with <code>CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION</code>) to stop runaway
delegation loops; <code>/clear</code> resets the budget</li>
<li>MCP tool calls running longer than 2 minutes now move to the
background automatically so the session stays usable; configure the
threshold or disable with
<code>CLAUDE_CODE_MCP_AUTO_BACKGROUND_MS</code></li>
<li>Typing <code>/resume</code> in the agent view now opens a picker of
past sessions — including sessions deleted from the list — and resumes
your pick as a background session</li>
<li>Fixed plan mode auto-running file-modifying Bash commands (e.g.
<code>touch</code>, <code>rm</code>) without a permission prompt or SDK
<code>canUseTool</code> callback</li>
<li>Fixed worktree creation following a repository-committed symlink at
<code>.claude/worktrees</code>, which could create files outside the
repository</li>
<li>Fixed a <code>continue:false</code> hook's halt being dropped when
the tool fails or completes mid-stream, and hook infrastructure errors
being misreported as user rejections</li>
<li>Fixed SIGTERM during a running Bash tool orphaning the command's
process tree in print/SDK mode; the CLI now aborts the turn, kills the
tree, and exits 143</li>
<li>Fixed <code>/background</code> and <code>claude --bg</code> failing
with "EUNKNOWN: unknown error, uv_spawn" on Windows when Group
Policy blocks PowerShell 5.1; the daemon now prefers PowerShell 7</li>
<li>Fixed shell mode (<code>!</code>) not executing commands containing
file paths while the path autocomplete popup was open</li>
<li>Fixed auto-mode denial notifications rendering broken characters
when a long denial reason was truncated mid-emoji</li>
<li>Fixed Ctrl+J not inserting a newline in the agent view dispatch
input on terminals with extended key reporting, and surfaced the newline
shortcut in the <code>?</code> help overlay</li>
<li>Fixed <code>/ultrareview</code> rejecting PR references like
<code>[#123](https://github.com/anthropics/claude-code/issues/123)</code>,
<code>PR 123</code>, and pasted PR URLs; error hints now name the
command you actually typed</li>
<li>Fixed <code>/ultrareview <branch></code> not fetching the
branch from origin when it exists remotely; it now suggests the closest
branch name on typos</li>
<li>Fixed <code>/ultrareview</code> skipping the billing confirmation in
a new conversation after <code>/clear</code></li>
<li>Fixed <code>/ultrareview</code>'s "not a git repository"
error on Claude Desktop now suggesting the project's repository folder
instead of terminal commands</li>
<li>Fixed hosted (host-managed) sessions failing at startup when
repository settings configured mTLS certs, extra CA bundles, or OAuth
scopes; these transport settings are now ignored with a warning</li>
<li>Fixed a spurious "File has not been read yet" error when
editing a file that had been read with offset/limit before resuming a
session</li>
<li>Fixed <code>ExitWorktree</code> failing with "no active
EnterWorktree session" after resuming a session with
<code>--continue</code>/<code>--resume</code> in print/SDK mode</li>
<li>Fixed the workflow agent grid staying empty for Remote Control
clients that join a session mid-run</li>
<li>Fixed streaming-mode control requests being marked complete before
their handler finished, which could lose the request on session
restart</li>
<li>Fixed background sessions created with <code>/fork</code> losing
their live-parent protection after a state write failure</li>
<li>Fixed reopening a stopped background session from the agent view
failing silently — it now resumes the session, or shows why it can't and
lets you force a restart</li>
<li>Fixed agent teams: a stopping teammate could send the leader
duplicate idle notifications when team initialization re-ran within a
session</li>
<li>Fixed the plan-approval dialog footer splitting "ctrl+g to edit
in <!-- raw HTML omitted -->" apart when the file path is long</li>
<li>Fixed the welcome banner keeping its old panel widths after a
combined width+height terminal resize in fullscreen mode</li>
<li>Fixed diff previews losing their line numbers and +/- markers in
narrow layouts</li>
<li>Fixed @-mentions attaching nothing after a partial file read, plugin
uninstall targeting the wrong marketplace, and false "Command timed
out" on exit code 143</li>
<li>Fixed OpenTelemetry HTTP exports being rejected with 411/400 by
Azure Monitor and other endpoints that don't accept chunked transfer
encoding</li>
<li>Fixed OTLP event log records missing
<code>trace_id</code>/<code>span_id</code> when <code>TRACEPARENT</code>
is set in SDK/headless mode</li>
<li>Fixed conversations with many images incorrectly failing with
"Request too large" errors, and improved the error message to
explain the actual cause</li>
<li>Fixed web search and web fetch returning "API Error" text
as search results or page content when the API was overloaded</li>
<li>Improved web search and web fetch reliability by retrying 529 errors
and rate-limited requests with bounded backoff</li>
<li>Improved prompt caching: the mid-conversation system block now works
behind LLM gateways and custom base URLs (Bedrock, Vertex, 1P)</li>
<li>Improved background agent attach: cold-attaching now instantly shows
the formatted transcript while the session boots, instead of a blank
wait</li>
<li>Reduced token usage in inter-agent messaging:
<code>SendMessage</code> bodies are no longer duplicated into replayed
history and tool results</li>
<li>Changed <code>/fork</code> to name the copy after your prompt when
the session has no title, so the row is recognizable in the agent
view</li>
<li>Changed bare <code>/btw</code> to reopen the side-question panel on
your most recent exchange so you can browse earlier answers</li>
<li>Changed the <code>←</code> footer hint to pulse <code>N done</code>
for a moment when a background agent finishes while nothing needs your
input</li>
<li>Deprecated the Task tool's <code>mode</code> parameter (now
ignored); subagents inherit the parent session's permission mode by
default</li>
<li>Changed Enterprise <code>forceLoginMethod</code> to be enforced for
VS Code extension, SDK, <code>setup-token</code>, and
<code>install-github-app</code> logins, not just the terminal</li>
<li>Changed session transcripts to record the reasoning effort level on
each assistant message</li>
<li>Changed headless/SDK sessions to apply a <code>set_model</code>
control request mid-turn; the next model round-trip uses the new model
instead of waiting for the next turn</li>
<li>Changed agent view / <code>claude agents --json</code>: sessions
waiting on a sandbox, MCP-input, or managed-settings prompt now show as
"Needs input" instead of "Working"</li>
<li>Updated the auth status panel title from "Cloud
authentication" to "Authentication"</li>
<li>Corrected an earlier release note (2.1.200): tmux through the 3.6
series lacks synchronized output; newer tmux with support is detected
automatically</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/anthropics/claude-code/commit/67f390c9a0b1440d369aebe2ff6a5023db35bf8e"><code>67f390c</code></a>
chore: Update CHANGELOG.md and feed.xml</li>
<li><a
href="https://github.com/anthropics/claude-code/commit/c39cb0f14bfe8bb519bae5bfc55add6867c5e2ab"><code>c39cb0f</code></a>
chore: Update CHANGELOG.md and feed.xml</li>
<li>See full diff in <a
href="https://github.com/anthropics/claude-code/compare/v2.1.210...v2.1.212">compare
view</a></li>
</ul>
</details>
<br />
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Author rich-form
evals/evals.jsonfor five judgment-bearing skills, one file per skill underplugins/<plugin>/skills/<skill>/evals/. Each ships 6 cases covering trigger/routing, the happy path, at least one refusal/guardrail, and at least one must-not-do anti-pattern, with objectively-verifiableexpectations.discovery/explorediscovery/explore-deepdiscovery/researchdiscovery/research-deepclaude-config-audit/memory-healthVariant skills exercise their distinctive contract rather than re-testing the parent:
CLAUDE_CODE_FORK_SUBAGENTunset, empty-scope handling, sidecar when EXPLORE.md exists.Per-skill warrant verdicts (re-checked against live SKILL.md)
No skill in this batch was pure-reference; no skip verdicts.
Version bumps
Both touched plugins bumped
0.1.0→0.2.0(minor — evals are an additive shipped component; a version bump is the only delivery vehicle). Matches the precedent of a minor bump for adding a new shipped component (implementation plugin → v0.2.0 for a setup skill, #100).Validation
check-jsonschemaagainstplugins/skill-quality/reference/evals.schema.json.check-skill.sh(0 errors); no evals-missing WARN, confirming presence is detected. Residual WARNs are pre-existing skill-body items (Gotchas surface /Use when:phrasing), out of scope for this evals-authoring task.Note on stale issue-body references
The claimed issue body cites two paths that do not exist on
main:docs/MIGRATION-PLAYBOOK.md"Evals — warrant policy and consumer-verify recipe" anddocs/evals-coverage.md. The warrant guidance actually lives inplugins/skill-quality/skills/skill-quality/SKILL.md(validate-evals action) — used as the warrant authority here. Flagging for tracker hygiene; not a blocker.Refs melodic-software/medley#1448
Note
Low Risk
Additive eval JSON and minor plugin version bumps only; no changes to skill runtime logic or application code paths.
Overview
Adds rich-form
evals/evals.json(six cases each, withprompt,expected_output, and verifiableexpectations) for five judgment-bearing skills: discoveryexplore,explore-deep,research,research-deep, and claude-config-auditmemory-health.The new cases encode each skill’s behavioral contract—happy paths, scope/routing, guardrails (read-only, no auto-fix, no training-recall claims), and variant-specific rules (e.g. fork isolation and sidecar artifacts for
explore-deep; dispatcher tiering and parallel topic agents forresearch-deep; contributor-local reports and fix gating formemory-health).Plugin versions are bumped as the delivery vehicle for the new shipped eval assets:
claude-config-audit0.1.0 → 0.2.0,discovery0.2.0 → 0.3.0.Reviewed by Cursor Bugbot for commit 9406015. Bugbot is set up for automated code reviews on this repo. Configure here.