You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
New codebase-audit plugin (category: quality) migrated from medley's in-repo .claude/skills/codebase-audit/. Verifies a codebase's factual claims — in docs, config, code,
and architecture notes — against reality, distinct from diff/PR review (judges a change) and from
Claude Code config audits (check settings.json/hooks/permissions).
Two skills, one cohesive capability:
/codebase-audit:codebase-audit — the eight-phase audit: prime the repo's conventions, discover
via one-subagent-per-file claim-extraction fan-out (scope-gated), independently validate each finding
(a separate agent re-verifies — never self-review), severity-rate in the required table with a
verified-non-issues proof-of-thoroughness list, then fix in priority order or stop at --review-only.
/codebase-audit:setup — re-runnable interview that infers audit targets from the repo layout and
writes the tracked .claude/codebase-audit.md config.
The in-repo skill hardcoded this repo's audit targets (libs/dotnet/Platform.*, Directory.Build.props, PLAT001-PLAT022, .claude/rules/**, a medley enumeration script). All of it is removed:
Audit-dimension seam (contract v2.1 seam 2). Per-dimension primary-sources / verification-sources / example-claims now resolve from a tracked .claude/codebase-audit.md
config, additive across ~/.claude/codebase-audit.md (user-global) → .claude/codebase-audit.md
(team) → .claude/codebase-audit.local.md (gitignored overlay). Four bundled dimensions
(documentation/configuration/code-quality/architecture) are tunable/removable; custom dimensions add.
Convention-resolution ladder. Config present → use; absent → infer from repo layout and offer to
persist via /codebase-audit:setup; cannot infer → ask + offer persist; else safe generic default.
No baked repo layout, ever.
Setup action. Ships the re-runnable setup skill that interviews + writes the tracked config
(the rejected Thariq config.json first-run pattern is not used).
Medley enumeration script (tools/codebase-audit/list-audit-targets.sh) dropped — targets come from
the config seam, not a bundled script. Adjacent-lane references generalized to /claude-config-audit:* "when installed", with inline out-of-scope fallback. Phase 4/5 build/verify/
simplify steps generalized to the consumer's own gates.
Gate evidence
claude plugin validate --strict ./plugins/codebase-audit — PASS
claude plugin validate --strict . (catalog manifest) — PASS
--plugin-dir smoke test in a clean non-medley repo (fresh git init scratch dir): headless /codebase-audit:setup load confirmed under namespace; replied SMOKE-OK codebase-audit.
claude plugin details: always-on ~319 tok (codebase-audit ~160 / setup ~160); on-invoke
~3.9k / ~1.1k.
Code execution — no hooks, no bundled scripts; markdown-only skill content plus per-skill evals/evals.json fixtures (established marketplace pattern).
MCP servers — none.
userConfig — none; audit targets flow through the tracked config file, no secrets anywhere.
Cache isolation — no ../ reach-outs, no absolute paths; templates referenced via ${CLAUDE_PLUGIN_ROOT}; the audit reads/writes only the consumer's own files under the given scope.
Data egress — none of its own; Phase 2 may use whatever documentation-research tools the
consumer's setup provides.
Provenance — first-party, lifted from melodic-software/medley and de-coupled per the migration
playbook.
Medium Risk
Markdown-only plugin with no hooks or bundled scripts, but a non–review-only run can modify consumer repo files and spawn many subagents; scope gates mitigate cost but depend on skill adherence.
Overview
Adds a new codebase-audit marketplace plugin (category: quality) and registers it in marketplace.json and the root README catalog.
The plugin ships two skills: /codebase-audit:codebase-audit (eight-phase claim verification with per-file subagent fan-out, independent re-validation, scope/cost gates, --review-only and dimension flags, optional fix/verify/review phases) and /codebase-audit:setup (interactive interview that writes tracked .claude/codebase-audit.md with layered overlays). Audit targets are repo-agnostic via configurable dimensions (documentation, configuration, code-quality, architecture) and merge semantics across user/team/local config — no bundled medley-specific paths or enumeration scripts.
Supporting markdown includes discovery method, category playbook, audit checklist, config template, phase checklist, and evals for both skills.
Reviewed by Cursor Bugbot for commit 5e3f53c. Bugbot is set up for automated code reviews on this repo. Configure here.
Check security posture and CLAUDE.md design rules compliance
Post findings
Summary: Well-designed plugin with clean de-coupling, strong security posture, and thorough behavioral contracts. One confirmed bug (missing ${CLAUDE_PLUGIN_ROOT} on context/reference paths) and a few design clarifications worth addressing before merge.
Bug: Missing ${CLAUDE_PLUGIN_ROOT} on context/reference file paths
This repo's established pattern (used in diagnose, review-toolkit, prototype) is to reference plugin-internal files with the ${CLAUDE_PLUGIN_ROOT} variable so the model can resolve them from the plugin cache directory in a consumer repo. The codebase-audit skill uses bare relative paths, which won't resolve in a consumer repo context.
# Current (broken in consumer context):
in [context/discovery-method.md](context/discovery-method.md)# Should match the established pattern (see diagnose/SKILL.md):
in [`${CLAUDE_PLUGIN_ROOT}/skills/codebase-audit/context/discovery-method.md`](context/discovery-method.md)
The checklist.md reference already correctly uses ${CLAUDE_PLUGIN_ROOT}/skills/codebase-audit/templates/checklist.md (SKILL.md:86-88) — apply the same pattern everywhere.
SKILL.md:60-71 and README.md:40-49 both describe the three-layer resolution as "additive" but don't define what happens when the same dimension name appears in two layers. Does the later layer replace the dimension's globs, append to them, or is it an error? A consumer who defines documentation both in ~/.claude/codebase-audit.md (user-global base) and in .claude/codebase-audit.md (team) needs to know whether they get the union of sources or an override. This gap is particularly sharp for example-claims, where deduplication semantics matter. Worth at least a one-sentence behavior spec in the SKILL.md dimension section or the config template header.
Tension: Phase 2 external research "required" vs. self-contained claim
SKILL.md:143-153 marks external research as "required, not optional" for API/library/best-practice claims. But the graceful-degrade section at SKILL.md:43-54 says "This skill is self-contained." A consumer without MCP docs servers or web search tools can't satisfy "required." The graceful-degrade section should either specify the fallback for Phase 2 (e.g., confidence-tag as needs-review when external verification is unavailable) or the "required" label should be softened to "strongly preferred."
Minor: Phase 7 mislabeled in model auto-invoke default
Phase 7 is Retrospective, not auto-fix. Should read "Phases 4–6 (auto-fix) and Phase 7 (retrospective) run only when..." or simply "Phases 4–7 run only when..."
Minor: Shell pipeline || echo "clean" doesn't fire for a clean tree
Working tree status: !`git status --porcelain 2>/dev/null | head -20 ||echo"clean"`
When the working tree is genuinely clean, git status --porcelain exits 0 with no output and head -20 also exits 0 — so || echo "clean" never fires. The label shows blank, not "clean." The fallback only catches the case where git itself is unavailable. Either remove || echo "clean" or use || echo "(unavailable)" to be accurate. The diagnose plugin uses || echo "none" as the fallback pattern consistently.
evals/evals.json:21-35 — the review-only-gate-stops eval checks that fixes aren't applied, but doesn't assert the required Verified non-issues section. SKILL.md Phase 3 mandates it even in --review-only mode. This is the thoroughness proof and a key behavioral contract worth asserting in the eval.
Minor: Background/unattended variant is underdefined
context/discovery-method.md:66-70 briefly mentions a "saved background workflow" form as an alternative executor. Nothing in the plugin implements or defines this — no saved workflow file, no instructions for creating one. Either remove this paragraph (nothing is lost; the in-session fan-out is the only defined form) or add it to a follow-up tracking item. As written it reads as scope-creep that could mislead consumers.
What looks good
Manifest is schema-valid and consistent with the existing plugin fleet (review-toolkit, diagnose — same shape, no extra fields needed).
Security posture is clean: no hooks, no MCP servers, no bundled scripts, no absolute paths, no secrets surface, ${CLAUDE_PLUGIN_ROOT} used where appropriate, consumer files are read/written only within scope.
Repo-agnostic design: config ladder (user-global → team → local overlay), convention-resolution ladder, and graceful inference are solid.
Cost gate on unscoped runs (eval chore: initialize marketplace scaffold #1) correctly models the risk — this is the most important behavioral contract and it's well-specified.
Category playbook severity vocabulary (error/warning/info, never HIGH/MEDIUM/LOW) is correct and consistent.
Evals cover the five critical behavioral branches: cost gate, review-only stop, config resolution, output format contract, no-invented-findings.
Marketplace entry and README catalog row are consistent and correctly formatted.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: low. Not approving because Cursor Bugbot skipped (usage limit reached) and did not complete automated review. Human review is needed; no reviewers were assigned because no non-author maintainers are available on this repository.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Codex P2: cross-file dimensions missing peer files in verification-sources
✅ Addressed in setup/SKILL.md defaults, config-template.md, and discovery-method.md fence note
One remaining minor observation
Phase 3 zero-findings path vs. --review-only gate — ambiguous ordering.
SKILL.md:211–220 (zero-findings outcome) says "Skip Phases 4-6. Proceed directly to Phase 7 (Retrospective)." But the --review-only gate two lines later says "present the full report and STOP." In a zero-findings --review-only run, both instructions apply and they conflict — the model has to pick one.
The checklist template (templates/checklist.md:18) disambiguates cleanly in favor of STOP: "Phases 4-7 SKIPPED in --review-only mode." So the checklist wins here, but the skill prose is slightly inconsistent. A one-word addition would close it:
- - Skip Phases 4-6 (nothing to fix). Proceed directly to Phase 7 (Retrospective) with scope/coverage- observations.+ - Skip Phases 4-6 (nothing to fix). If not `--review-only`, proceed to Phase 7 (Retrospective) with+ scope/coverage observations.
This is cosmetic — the checklist already gives the right answer — so not a blocker. Worth fixing in a follow-up or bundling with the next change.
All ${CLAUDE_PLUGIN_ROOT} references are consistent — every internal skill file path uses the variable; markdown link targets are kept as relative paths for human browsability. Pattern matches diagnose and review-toolkit.
Merge semantics spec is precise and covers the important edge cases: UNION for globs, concat+dedupe for claims, empty = opt-out (not silent inheritance).
Graceful degrade for missing research tools is well-defined: needs-review tag surfaces the gap for human judgment without dropping or asserting.
Cross-file dimension fence note in discovery-method.md (new paragraph, lines 63–70) is one of the highest-value additions — it's the root-cause explanation for why verification-sources must include source roots, not just a rule to follow. The setup defaults and config-template comments reinforce it.
Evals now cover all five critical behavioral contracts including verified-non-issues in review-only mode.
Security posture is clean: no hooks, no MCP, no scripts, no absolute paths, ${CLAUDE_PLUGIN_ROOT} only, consumer files scoped to audit target.
Manifest and marketplace entry are schema-valid and consistent with the plugin fleet.
Verdict: ready to merge. The fix commit is comprehensive and well-targeted. The zero-findings/--review-only nit is the only open item and it's not a behavioral regression — the checklist already carries the right answer.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: medium. Not approving because Cursor Bugbot skipped (usage limit reached) and did not complete automated review. Human review is needed; no reviewers were assigned because no non-author maintainers are available on this repository.
Sent by Cursor Approval Agent: Pull Request Router and Approver
The commit addresses the last observation from round 2 — the zero-findings/--review-only ordering ambiguity in Phase 3. The single-line change is exactly right:
- Skip Phases 4-6 (nothing to fix). Proceed directly to Phase 7 (Retrospective) with scope/coverage- observations.+ Skip Phases 4-6 (nothing to fix). If not `--review-only`, proceed to Phase 7 (Retrospective) with+ scope/coverage observations.
This now matches templates/checklist.md:18 ("Phases 4-7 SKIPPED in --review-only mode") with no residual conflict.
One open item: Codex P2 — checklist write in --review-only mode (minor)
The Codex review raised a valid concern that wasn't addressed in the fix commit. The "Emit checklist" instruction at SKILL.md:91–96 currently reads:
For any audit run (Phases 0-7), copy … into wherever the consuming repo keeps working task notes (or keep it in-response).
The parenthetical (or keep it in-response) is the escape hatch, but the instruction applies unconditionally and doesn't name --review-only as the condition that mandates it. Eval #2 (review-only-gate-stops) explicitly expects "Does not modify, edit, or write any files" — a behavioral contract the current prose enforces only implicitly. A model following the instruction literally could copy the checklist to disk before reaching the Phase 3 stop gate.
Recommended one-line addition:
For any audit run (Phases 0-7), copy
`${CLAUDE_PLUGIN_ROOT}/skills/codebase-audit/templates/checklist.md` into wherever the consuming
-repo keeps working task notes (or keep it in-response). Tick each phase as completed. Phases 4-7 may-SKIP per `--review-only` mode.+repo keeps working task notes (or keep it in-response — in `--review-only` mode, always keep+in-response, never write to disk). Tick each phase as completed. Phases 4-7 SKIPPED in+`--review-only` mode.
Not a blocker — the escape hatch exists — but it leaves the no-writes contract only weakly stated for the one file write that precedes the stop gate.
The round-2 fix commit closed every prior finding cleanly; the round-3 commit (d6b8638) closes the only remaining nit.
All prior review confirmations still hold: ${CLAUDE_PLUGIN_ROOT} references consistent, merge semantics precise, graceful-degrade well-defined, evals cover five critical behavioral branches, security posture clean.
Phase 3 zero-findings and --review-only gate now agree — templates/checklist.md, SKILL.md Phase 3 prose, and the Model auto-invoke default section are all in sync.
Verdict: nearly merge-ready. The Codex P2 is the only open item and it's not a regression — the existing (or keep it in-response) already gives a conforming model the right path. Worth closing before merge but not a blocker if the PR author prefers to land this and follow up.
The reason will be displayed to describe this comment to others. Learn more.
Stale comment
Risk: medium. Not approving because Cursor Bugbot skipped (usage limit reached) and did not complete automated review. Human review is needed; no reviewers were assigned because no non-author maintainers are available on this repository.
Sent by Cursor Approval Agent: Pull Request Router and Approver
Author + ship the codebase-audit plugin (category: quality) from medley's
in-repo skill. Verifies factual claims in docs/config/code/architecture
against reality via per-file subagent fan-out with independent validation.
De-couples all medley path/convention assumptions: audit dimensions and
their primary/verification/example-claim targets now flow through a tracked
.claude/codebase-audit.md config seam (extensibility contract v2.1 seam 2,
additive user-global -> team -> local overlay resolution), written by a
re-runnable /codebase-audit:setup skill per the convention-resolution ladder.
Refs melodic-software/medley#1376
- Use ${CLAUDE_PLUGIN_ROOT} on all internal skill file references so they
resolve from the plugin cache in a consumer repo (matches diagnose/
review-toolkit pattern; checklist ref already did).
- Spec additive merge semantics when a dimension name spans config layers
(union globs, concat+dedupe example-claims, empty lists = explicit opt-out).
- Cross-file dimensions (code-quality/architecture) must list source roots +
manifests in verification-sources so discovery agents can read peer files
under the read-only fence — wired in setup defaults, template, and the
discovery fence note (Codex P2).
- Soften Phase 2 "required" to "required when tooling exists" + needs-review
degrade path for missing research tools.
- Fix Phase 7 label (retrospective, not auto-fix); precomputed-context clean-
tree fallback (blank vs misleading "clean"); add verified-non-issues eval
expectation; drop the underdefined background-workflow paragraph.
Refs melodic-software/medley#1376
Verify the 5e3f53c fix against the round 3 remaining observation
Confirm status of open Codex P2 (checklist write in --review-only mode)
Post findings
5e3f53c fix: confirmed clean
This commit closes the round-2 cosmetic nit that was still noted as outstanding in the round-3 review — the zero-findings outcome in Phase 3 unconditionally proceeding to Phase 7 even in --review-only mode:
- Skip Phases 4-6 (nothing to fix). Proceed directly to Phase 7 (Retrospective) with scope/coverage- observations.+ Skip Phases 4-6 (nothing to fix). If not `--review-only`, proceed to Phase 7 (Retrospective) with+ scope/coverage observations.
Exactly the one-word addition recommended in round 2 and flagged as the final cosmetic nit in round 3. The prose now agrees with templates/checklist.md:18 ("Phases 4-7 SKIPPED in --review-only mode") with no residual conflict.
One open item: Codex P2 — checklist write in --review-only mode (minor, unchanged from round 3)
For any audit run (Phases 0-7), copy
`${CLAUDE_PLUGIN_ROOT}/skills/codebase-audit/templates/checklist.md` into wherever the consuming
repo keeps working task notes (or keep it in-response). Tick each phase as completed. Phases 4-7 may
SKIP per `--review-only` mode.
Two issues remain:
Weak file-write gate — the no-disk-write contract for --review-only runs is only implied by the (or keep it in-response) escape hatch. The eval (review-only-gate-stops) explicitly asserts "Does not modify, edit, or write any files" — a model following the "copy into task notes" instruction literally could write the checklist before reaching Phase 3's STOP gate.
Inconsistent wording — "may SKIP" here vs "SKIPPED" in templates/checklist.md:18. The template is authoritative but the prose is softer.
Recommended one-line tightening:
repo keeps working task notes (or keep it in-response — in `--review-only` mode, always keep
-in-response, never write to disk). Tick each phase as completed. Phases 4-7 may-SKIP per `--review-only` mode.+in-response, never write to disk). Tick each phase as completed. Phases 4-7 SKIPPED in+`--review-only` mode.
Phase 2 graceful degrade well-specified (needs-review tag when external research unavailable)
Phase 3 zero-findings and --review-only gate now fully agree with the checklist template
Evals cover all five critical behavioral contracts
Security posture clean: no hooks, MCP, scripts, absolute paths, or egress
Manifest and marketplace entry schema-valid and consistent with the plugin fleet
Verdict: nearly merge-ready. The Codex P2 (checklist prose weak on --review-only writes) is the only open item and is not a behavioral regression — the eval and escape hatch already carry the right answer. Worth addressing before merge or in a follow-up.
The reason will be displayed to describe this comment to others. Learn more.
Risk: medium. Not approving because Cursor Bugbot skipped (usage limit reached) and did not complete automated review. Human review is needed; no reviewers were assigned because no non-author maintainers are available on this repository.
Sent by Cursor Approval Agent: Pull Request Router and Approver
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs melodic-software/medley#1376
What
New
codebase-auditplugin (category: quality) migrated from medley's in-repo.claude/skills/codebase-audit/. Verifies a codebase's factual claims — in docs, config, code,and architecture notes — against reality, distinct from diff/PR review (judges a change) and from
Claude Code config audits (check
settings.json/hooks/permissions).Two skills, one cohesive capability:
/codebase-audit:codebase-audit— the eight-phase audit: prime the repo's conventions, discovervia one-subagent-per-file claim-extraction fan-out (scope-gated), independently validate each finding
(a separate agent re-verifies — never self-review), severity-rate in the required table with a
verified-non-issues proof-of-thoroughness list, then fix in priority order or stop at
--review-only./codebase-audit:setup— re-runnable interview that infers audit targets from the repo layout andwrites the tracked
.claude/codebase-audit.mdconfig.Marketplace entry appended to
.claude-plugin/marketplace.json(category: quality); README catalogrow appended.
plugin.jsonpinsversion: 0.1.0.De-coupling from the source repo (contract v2.1)
The in-repo skill hardcoded this repo's audit targets (
libs/dotnet/Platform.*,Directory.Build.props,PLAT001-PLAT022,.claude/rules/**, a medley enumeration script). All of it is removed:primary-sources/verification-sources/example-claimsnow resolve from a tracked.claude/codebase-audit.mdconfig, additive across
~/.claude/codebase-audit.md(user-global) →.claude/codebase-audit.md(team) →
.claude/codebase-audit.local.md(gitignored overlay). Four bundled dimensions(documentation/configuration/code-quality/architecture) are tunable/removable; custom dimensions add.
persist via
/codebase-audit:setup; cannot infer → ask + offer persist; else safe generic default.No baked repo layout, ever.
setupskill that interviews + writes the tracked config(the rejected Thariq
config.jsonfirst-run pattern is not used).tools/codebase-audit/list-audit-targets.sh) dropped — targets come fromthe config seam, not a bundled script. Adjacent-lane references generalized to
/claude-config-audit:*"when installed", with inline out-of-scope fallback. Phase 4/5 build/verify/simplify steps generalized to the consumer's own gates.
Gate evidence
claude plugin validate --strict ./plugins/codebase-audit— PASSclaude plugin validate --strict .(catalog manifest) — PASS--plugin-dirsmoke test in a clean non-medley repo (freshgit initscratch dir): headless/codebase-audit:setupload confirmed under namespace; repliedSMOKE-OK codebase-audit.claude plugin details: always-on ~319 tok (codebase-audit ~160 / setup ~160); on-invoke~3.9k / ~1.1k.
typos(plugin + touched files): clean.markdownlint-cli2(8 files): 0 errors. All JSON valid (jq).Security review (plugin-acceptance)
evals/evals.jsonfixtures (established marketplace pattern).../reach-outs, no absolute paths; templates referenced via${CLAUDE_PLUGIN_ROOT}; the audit reads/writes only the consumer's own files under the given scope.consumer's setup provides.
playbook.
🤖 Generated with Claude Code
Note
Medium Risk
Markdown-only plugin with no hooks or bundled scripts, but a non–review-only run can modify consumer repo files and spawn many subagents; scope gates mitigate cost but depend on skill adherence.
Overview
Adds a new
codebase-auditmarketplace plugin (category: quality) and registers it inmarketplace.jsonand the root README catalog.The plugin ships two skills:
/codebase-audit:codebase-audit(eight-phase claim verification with per-file subagent fan-out, independent re-validation, scope/cost gates,--review-onlyand dimension flags, optional fix/verify/review phases) and/codebase-audit:setup(interactive interview that writes tracked.claude/codebase-audit.mdwith layered overlays). Audit targets are repo-agnostic via configurable dimensions (documentation,configuration,code-quality,architecture) and merge semantics across user/team/local config — no bundled medley-specific paths or enumeration scripts.Supporting markdown includes discovery method, category playbook, audit checklist, config template, phase checklist, and evals for both skills.
Reviewed by Cursor Bugbot for commit 5e3f53c. Bugbot is set up for automated code reviews on this repo. Configure here.