You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adds rich-form evals/evals.json for five implementation skills — implement, implement-dispatch, build, lint, setup — and bumps the plugin manifest 0.2.0 → 0.3.0 so marketplace consumers pick up the new shipped eval component.
Skill
Cases
Focus
implement
6
bugfix-mode incremental cadence, branch-check stops on default branch, major divergence routes back to planning, never-declares-impossible-without-alternatives, over-correction guard on pushback, autonomous run chains to /implement-dispatch
implement-dispatch
6
autonomous dispatches even when plan is all-main-window, interactive all-main-window hands back to /implement, verify worker return against direct evidence, brief carries scope-fence + divergence clause, non-interactive moderate divergence takes conservative + DEVIATIONS.md, autonomous fresh-context verifier before DONE
check-mode lint-only detection, --fix mode parse, owns yaml+cross-cutting, missing-tool skip + file-outside-ecosystem silent, no-changes directs to /lint all
setup
4
infer-then-interview before writing, idempotent read-existing-first, writes consumer-tracked config not plugin dir, ambiguous-inference asks not guesses
Each case carries id, kebab-case name, prompt, expected_output, files, and an expectations array of objectively-verifiable checks — modeled on the bug-report rich-form exemplar. No fixtures needed (routing/discipline/guardrail behaviors).
Warrant
All five skills carry judgment-bearing behavioral contracts that could silently regress, so no skip verdicts — including the three the issue flagged as gray-zone (build/lint/setup):
build / lint — ecosystem-detection routing, skip-not-FAIL missing-tool guardrail, consumer-config-wins ladder precedence, and the build-vs-lint scope split (cross-cutting's ** must never leak into /build). These are emit-shape + guardrail contracts, not pure mechanics — cases target the judgment, not "does it detect dotnet".
setup — a config-writer skill is explicitly warrantable per the playbook (the codebase-audit/setup eval is the model): interview/infer/write-scope discipline, idempotent read-existing, and the write-only-in-consumer-.claude/ecosystems/ guardrail.
Validation
check-jsonschema against plugins/skill-quality/reference/evals.schema.json — all 5 conform.
check-skill.sh (skills_root = plugins/implementation/skills) — all 5 PASS, 0 errors. Remaining WARNs are pre-existing skill-body Gotchas-surface items, out of scope for an evals-only PR.
The issue left the level unspecified. implementation is one multi-skill plugin, so a single manifest bump covers all five; chose a minor bump (0.2.0 → 0.3.0) — evals are an additive, backward-compatible shipped component with no behavioral change to the skills. Matches the docs-hygiene evals-backfill precedent (#132); reversible to patch if a maintainer prefers.
Refs melodic-software/medley#1449
Note
Low Risk
Documentation and test-artifact JSON only; no runtime or skill logic changes.
Overview
Adds rich-formevals/evals.json for five implementation skills (implement, implement-dispatch, build, lint, setup) so judgment-bearing contracts can be regression-tested with prompts, expected_output, and verifiable expectations (no fixtures). Coverage spans orchestration routing and divergence guards, worker verification and scope-fenced briefs, build/lint ecosystem scope and skip-not-FAIL behavior, consumer config ladder precedence, and setup interview/write-location discipline.
Bumps the plugin manifest 0.2.0 → 0.3.0 and documents the release in CHANGELOG as an additive shipped component with no skill behavior changes.
Reviewed by Cursor Bugbot for commit 26d367d. Bugbot is set up for automated code reviews on this repo. Configure here.
Add rich-form evals/evals.json for implement, implement-dispatch, build,
lint, and setup, and bump the implementation plugin 0.2.0 -> 0.3.0 so
marketplace consumers pick up the new shipped eval component.
All five skills warrant evals (no skip verdicts): each carries a
judgment-bearing behavioral contract that could silently regress —
implement (feature/fix/refactor/config mode routing, divergence
detection, scope-fence over-correction guard, never-declare-impossible),
implement-dispatch (autonomous-vs-interactive orchestration routing,
scope-fenced briefs, verify-against-direct-evidence, conservative
non-interactive divergence + DEVIATIONS.md, fresh-context verifier),
build and lint (ecosystem-detection routing, skip-not-FAIL missing-tool
guardrail, consumer-config-wins precedence, build-vs-lint scope split),
and setup (infer-then-interview, idempotent read-existing, write-scope
guardrail, ambiguous-inference-asks). Cases are modeled on the
bug-report rich-form exemplar and validated against
plugins/skill-quality/reference/evals.schema.json.
Refs melodic-software/medley#1449
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read skill SKILL.md files for all 5 covered skills
Review against evals schema and project conventions
Post findings
Summary
The five evals files are accurate, well-formed, and cover the right contracts. Every expectation I traced back to its source SKILL.md is grounded in documented behavior — no invented or over-promised claims. Schema compliance is confirmed by the PR description's validation steps. No blockers. Three observations below, all minor.
Findings
Observation 1 — implement/evals.json case 1 prompt omits the slash command (intentional, but worth flagging for eval runners)
"prompt": "There's a bug in the login flow — expired tokens are being accepted. Fix it."
All other cases in this file use the explicit /implementation:implement prefix. This case tests that the skill detects Bugfix mode from the natural-language 'fix' signal (the disable-model-invocation: false path). That's a valid and important scenario to cover.
However, when evaluators follow the playbook's manual-exercise recipe ("paste its prompt into a fresh session"), this case requires the skill to fire as a model-invoked skill rather than a user-invoked one — which is less predictable than the other five cases. A one-line note in expected_output like "(invoked as a model-invoked skill from natural language — no explicit /implement prefix)" would help eval runners understand the intent rather than wondering if the slash command was accidentally omitted.
Not a correctness issue; a readability hint.
Observation 2 — build/evals.json case 4, expectation 3 slightly over-specifies
"Does NOT run the `cross-cutting` surface (its `**` glob is never matched by /build); points the user to /lint for those"
The SKILL.md confirms that /build never matches the cross-cutting** glob and won't run yaml for the workflow file. But the SKILL.md doesn't explicitly promise a redirect suggestion to /lint when those surfaces are encountered in a /build all run — it just silently excludes them. The /build description does say "for lint-only use /lint" so the redirect is reasonable expected behavior, but it isn't specified in the workflow steps.
The expected_output prose ("points the user to /lint for those") reads correctly; it's only this specific expectation item that asserts it as a verifiable check. Worth tightening to "Does NOT run yaml or cross-cutting surfaces; those are /lint-only" if the redirect isn't guaranteed by the skill contract.
Keep a Changelog format calls for ## [0.3.0] - YYYY-MM-DD. If the existing ## [0.2.0] entry also lacks a date this is consistent with the established convention in this file and is not a concern. Only flag if earlier entries have dates.
What's solid
Accuracy: every expectation traces cleanly to a documented SKILL.md contract. The autonomy/interactive mode split in implement-dispatch cases 1 and 2, the scope-fence + divergence-escalation clause in case 4, the conservative-option + DEVIATIONS.md in case 5, and the fresh-context verifier gate in case 6 all match the SKILL.md exactly.
Coverage per playbook: all five skills hit trigger/routing, happy-path, guardrail, and anti-pattern cases as the playbook recommends.
Warrant: the three gray-zone skills (build, lint, setup) are correctly warranted — all three carry judgment-bearing emit-shape and guardrail contracts, not pure mechanics.
Schema conformity: all five files satisfy the required skill_name / evals[] / id / prompt fields; name fields are properly kebab-case; expectations items are strings throughout, consistent with other rich-form exemplars.
setup evals: closely model the codebase-audit/setup exemplar the playbook names as the reference — three cases parallel exactly (infer-then-interview, idempotent-read-existing, writes-consumer-tracked), with a fourth that covers the ambiguous-inference branch that codebase-audit/setup doesn't.
Version bump: minor (0.2.0 → 0.3.0) is the right level — additive, backward-compatible, no behavioral change to the skills themselves. Matches the docs-hygiene evals-backfill precedent.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving: Cursor Bugbot skipped (usage limit reached) and did not produce a clean review signal. Human review is needed before merge; no reviewers were assigned.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Two minor bot-review improvements (no behavioral change):
- implement case 1 expected_output now notes the case fires as a
model-invoked skill from natural language (no explicit slash prefix),
so eval runners understand the intent.
- build case 4 expectation 3 drops the unverifiable "/lint redirect"
assertion; the verifiable contract is that /build never runs the
yaml/cross-cutting surfaces.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thanks for the review. Addressed the two actionable observations in 3c84a44:
Obs 1 — implement case 1 expected_output now states it fires as a model-invoked skill from natural language (no slash prefix), clarifying intent for eval runners.
Obs 2 — build case 4 expectation 3 drops the unverifiable "/lint redirect" claim; the verifiable contract is that /build never runs the yaml/cross-cutting surfaces.
Obs 3 — no change: [0.2.0] and [0.1.0] carry no dates, so an undated [0.3.0] is consistent with this file's established convention.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving: Cursor Bugbot completed as skipped (usage limit reached) and did not provide a clean automated review signal. Human review is still needed before merge; no reviewers were assigned because the only non-bot contributor is the PR author.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Codex review (P2): build case 3 sets up a valid tracked dotnet.yaml but
one expectation graded malformed-file degrade behavior — a scenario
mismatch that could fail a correct valid-config answer. Replace that
expectation with a consumer-cmd-wins check, and cover the
malformed-degrades-to-inference guardrail in its own dedicated case 6.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving: Cursor Bugbot completed as skipped (usage limit reached) and did not provide a clean automated review signal. Human review is needed before merge; no reviewers were assigned because the only repository contributor is the PR author.
Sent by Cursor Approval Agent: Pull Request Router and Approver
…case
Codex review (P2): lint case 4 expected a missing-tool `skip` for
markdown, but /lint honors opt-in first — without a markdownlint config,
silently skipping markdown is the correct behavior, which could penalize
a valid answer. State in the prompt that the repo has opted into
markdownlint (config present) so the missing-tool skip is the
unambiguously-graded behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
Risk: low. Not approving: Cursor Bugbot completed as skipped (usage limit reached) and did not provide a clean automated review signal. Human review is needed before merge; no reviewers were assigned because the only repository contributor is the PR author.
Sent by Cursor Approval Agent: Pull Request Router and Approver
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds rich-form
evals/evals.jsonfor fiveimplementationskills —implement,implement-dispatch,build,lint,setup— and bumps the plugin manifest0.2.0→0.3.0so marketplace consumers pick up the new shipped eval component.implement/implement-dispatchimplement-dispatch/implement, verify worker return against direct evidence, brief carries scope-fence + divergence clause, non-interactive moderate divergence takes conservative +DEVIATIONS.md, autonomous fresh-context verifier before DONEbuildyaml/cross-cutting, clean-tree branch-diff fallbacklint--fixmode parse, ownsyaml+cross-cutting, missing-tool skip + file-outside-ecosystem silent, no-changes directs to/lint allsetupEach case carries
id, kebab-casename,prompt,expected_output,files, and anexpectationsarray of objectively-verifiable checks — modeled on thebug-reportrich-form exemplar. No fixtures needed (routing/discipline/guardrail behaviors).Warrant
All five skills carry judgment-bearing behavioral contracts that could silently regress, so no skip verdicts — including the three the issue flagged as gray-zone (
build/lint/setup):implement/implement-dispatch— mode/orchestration routing, divergence handling, scope-fence and over-correction guards.build/lint— ecosystem-detection routing,skip-not-FAILmissing-tool guardrail, consumer-config-wins ladder precedence, and the build-vs-lint scope split (cross-cutting's**must never leak into/build). These are emit-shape + guardrail contracts, not pure mechanics — cases target the judgment, not "does it detect dotnet".setup— a config-writer skill is explicitly warrantable per the playbook (thecodebase-audit/setupeval is the model): interview/infer/write-scope discipline, idempotent read-existing, and the write-only-in-consumer-.claude/ecosystems/guardrail.Validation
check-jsonschemaagainstplugins/skill-quality/reference/evals.schema.json— all 5 conform.check-skill.sh(skills_root =plugins/implementation/skills) — all 5 PASS, 0 errors. Remaining WARNs are pre-existing skill-body Gotchas-surface items, out of scope for an evals-only PR.scripts/validate-plugins.sh --strict(manifests + catalog) — pass.typosclean.Version-bump level
The issue left the level unspecified.
implementationis one multi-skill plugin, so a single manifest bump covers all five; chose a minor bump (0.2.0→0.3.0) — evals are an additive, backward-compatible shipped component with no behavioral change to the skills. Matches the docs-hygiene evals-backfill precedent (#132); reversible to patch if a maintainer prefers.Refs melodic-software/medley#1449
Note
Low Risk
Documentation and test-artifact JSON only; no runtime or skill logic changes.
Overview
Adds rich-form
evals/evals.jsonfor five implementation skills (implement,implement-dispatch,build,lint,setup) so judgment-bearing contracts can be regression-tested with prompts,expected_output, and verifiableexpectations(no fixtures). Coverage spans orchestration routing and divergence guards, worker verification and scope-fenced briefs, build/lint ecosystem scope and skip-not-FAIL behavior, consumer config ladder precedence, and setup interview/write-location discipline.Bumps the plugin manifest 0.2.0 → 0.3.0 and documents the release in CHANGELOG as an additive shipped component with no skill behavior changes.
Reviewed by Cursor Bugbot for commit 26d367d. Bugbot is set up for automated code reviews on this repo. Configure here.