feat(claude-config): add audit-model-fit unhobbling audit#809
Conversation
New fourth audit skill for the claude-config plugin. Sweeps local Claude Code instruction surfaces — user + project CLAUDE.md, skill SKILL.md bodies + context files, agent definitions, .claude/rules, prompt-type hooks and output styles — for deterministic constraints that hobble newer, more capable models, and proposes removals/rewrites. Check catalog (from the issue's sourced seeds): bare prohibitions with no rationale (rewrite to add the why, never blanket-delete), over-prescriptive step lists (cull to intent + hard constraints), over-constraining example blocks (trim toward 3-5, not a blanket ban), and stale model-era workarounds — each measured against "would removing this cause Claude to make mistakes?". A bundled instruction-surface-scan.sh enumerates the surfaces and flags the two grep-able smells (bare prohibitions, example-dense files) as candidates; the judgment stays in the skill body. Prose surfaces are grep-scanned; hook scripts are counted-only (their prompt text is embedded in code) and read by hand in Phase 2. Report-only and human-gated: it presents findings plus proposed diffs and never edits any instruction file itself (no --fix). Findings inside melodic-software/standards-managed materializations route upstream per the sync-manifest rather than being edited in place. Composes with (distinct intents, pointers only) claude-memory:audit, skill-quality:check, docs-hygiene:compress, and the sibling audit. Version bumped 0.7.1 -> 0.8.0; README + CHANGELOG updated to four skills. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1V3gkrfSf75isB8MiDy3o
|
Claude finished @kyle-sexton's task in 2m 35s —— View job Security Review — PR #809
ScopeReviewed all 8 changed files:
No GitHub Actions workflow files were changed. FindingsNo CRITICAL or IMPORTANT security vulnerabilities found. The two shell scripts are the primary attack surface; both are well-hardened:
SUGGESTION (low confidence — non-exploitable in practice)
hits="$(grep -icE "$PROHIBITION" "$f" 2>/dev/null)"A filename beginning with # before
grep -icE "$PROHIBITION" "$f"
# after
grep -icE "$PROHIBITION" -- "$f"This is a SUGGESTION only — not a blocker. SummaryThis PR is clean from a security standpoint. The shell code is defensively written, and the report-only/human-gated design of the skill eliminates the highest-risk outcome (unattended file mutation). Ready to merge. |
|
Claude finished @kyle-sexton's task in 2m 57s —— View job PR Review:
|
| Rule | Status |
|---|---|
| Repo-agnostic (no hardcoded paths/names) | ✅ Uses ${CLAUDE_PLUGIN_ROOT}, git rev-parse, $HOME |
| Configurable without editing the plugin | ✅ Consumer exceptions read from consuming repo's own files |
Plugin-form-safe (${CLAUDE_PLUGIN_ROOT} only) |
✅ |
| No PII/secrets | ✅ |
Versioned (0.7.1 → 0.8.0) |
✅ Minor bump correct for new skill |
| Plugin acceptance: no unjustified egress | ✅ No network calls; instruction-surface-scan.sh is local only |
Documentation / Changelog
CHANGELOG.md[0.8.0]entry is comprehensive and accurate.README.mdtable andaudit-model-fitsection are consistent with SKILL.md and criteria.md.- The clarification at README:19-21 ("
audit-permission-grantsandaudit-model-fitare report-only") correctly distinguishes them from the skills that have a--fix/--implementpath.
Summary
One non-blocking finding (doc/code inconsistency on don't), one style note (shell escape readability). Everything else is correct. The implementation matches the sibling pattern, is secure, and is safe to merge as-is. The don't discrepancy is worth a one-line fix before merging for documentation accuracy, but does not affect runtime behavior.
The root README.md plugin catalog is generated from each plugin.json description and gated for drift; bumping the claude-config description to "Four audit skills" left the root listing stale and failed plugin-gate. Regenerate-in-sync: node scripts/generate-catalog.mjs reports no drift. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1V3gkrfSf75isB8MiDy3o
|
Claude finished @kyle-sexton's task in 2m 30s —— View job Security Review — PR #809
ScopeReviewed all 9 changed files:
No GitHub Actions workflow files were changed. FindingsNo CRITICAL or IMPORTANT security vulnerabilities found. Shell script hardening — controls verified
|
|
Claude finished @kyle-sexton's task in 2m 26s —— View job PR Review:
|
| Rule | Status |
|---|---|
| Repo-agnostic (no hardcoded paths) | ✓ Uses ${CLAUDE_PLUGIN_ROOT}, git rev-parse, $HOME |
| Configurable without editing the plugin | ✓ Consumer exceptions read from consuming repo's own files |
Plugin-form-safe (${CLAUDE_PLUGIN_ROOT} only) |
✓ |
| No PII/secrets | ✓ |
Versioned (0.7.1 → 0.8.0) |
✓ Minor bump correct for new skill |
| No unjustified egress | ✓ No network calls; script is local-only |
| Security review cleared | ✓ No critical or important vulnerabilities |
Verdict
No correctness bugs, no security vulnerabilities. Two non-blocking findings remain open from prior reviews (the don't doc gap and the grep -- suggestion). Both are safe to merge as-is; the don't doc fix is the more meaningful cleanup if a follow-up commit is warranted.
The scripts were committed 100644 from a Windows worktree (no exec bit); the hygiene gate requires shebang scripts to be 100755, matching the sibling audit-* scripts. Set via git update-index --chmod=+x. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1V3gkrfSf75isB8MiDy3o
|
Claude finished @kyle-sexton's task in 2m 39s —— View job PR Review:
|
| Rule | Status |
|---|---|
| Repo-agnostic | ✓ Uses ${CLAUDE_PLUGIN_ROOT}, git rev-parse, $HOME |
| Configurable without editing the plugin | ✓ Consumer exceptions read from consuming repo's own files |
Plugin-form-safe (${CLAUDE_PLUGIN_ROOT} only) |
✓ No ../ reach-outs |
| No PII/secrets | ✓ |
Versioned (0.7.1 → 0.8.0) |
✓ Minor bump correct for a new skill |
| No unjustified egress | ✓ No network calls; script is local-only |
| Security review cleared | ✓ No critical or important vulnerabilities |
Verdict
No correctness bugs, no security vulnerabilities. The exec-bit fixup (c12ca86) resolves the hygiene gate failure that triggered this sync. Two non-blocking findings remain open from prior reviews — the don't doc gap is the more meaningful cleanup (one-line criteria.md edit); the grep -- suggestion is defensive-only. Both are safe to merge as-is.
|
Claude finished @kyle-sexton's task in 2m 21s —— View job Security Review — PR #809
ScopeReviewed all 9 changed files:
No GitHub Actions workflow files were changed. FindingsNo CRITICAL or IMPORTANT security vulnerabilities found. Shell script hardening — controls verified
|
|
Post-green review classification (work lane):
No DEFER-FORBIDDEN class applies to either open finding. No inline review threads exist on this PR (GraphQL query returned empty) — nothing to resolve. Handing off to the babysit lane for ongoing watch. |
|
Claude finished @kyle-sexton's task in 2m 19s —— View job PR Review:
|
| Control | Status |
|---|---|
set -u / set -uo pipefail in test |
✓ |
NUL-safe surface collection (find -print0 / read -r -d '') |
✓ |
PROHIBITION is a hardcoded literal — no user-controlled input reaches -E |
✓ |
printf '%s' for untrusted candidate_out accumulation |
✓ |
Bounded output (cut -c1-200, head -n 200) |
✓ |
2>/dev/null suppresses error disclosure |
✓ |
[[ -n "$repo_root" ]] guard before cd |
✓ |
HOME="$FIX" fixture isolation in tests |
✓ |
trap 'rm -rf "$FIX"' EXIT cleanup |
✓ |
!-injection emits only integer counts — no file content at load time, prompt-injection surface at skill-load is zero |
✓ |
| Report-only / human-gated design — no unattended file mutation risk | ✓ |
CLAUDE.md Alignment
| Rule | Status |
|---|---|
| Repo-agnostic — no hardcoded paths | ✓ Uses ${CLAUDE_PLUGIN_ROOT}, git rev-parse, $HOME |
| Configurable without editing the plugin | ✓ Consumer exceptions read from consuming repo's own files |
Plugin-form-safe (${CLAUDE_PLUGIN_ROOT} only) |
✓ No ../ reach-outs |
| No PII / secrets | ✓ |
Versioned (0.7.1 → 0.8.0) |
✓ Minor bump correct for a new skill |
| No unjustified egress | ✓ No network calls; script is local-only |
Open Findings
1. don't absent from the C1 keyword list in criteria.md — doc/code inconsistency (non-blocking)
The PROHIBITION regex at instruction-surface-scan.sh:56 includes don't, but criteria.md:51 lists only never, do not, must not, shall not. No runtime impact — don't is a legitimate bare prohibition form and the regex is correct. Carried from prior reviews; kyle-sexton classified as noted-not-fixed.
2. Same gap in the script's user-visible summary output — new (non-blocking)
Prior reviews caught the criteria.md gap but missed a second instance of the same inconsistency. The summary heredoc at instruction-surface-scan.sh:162 prints:
Bare-prohibition lines (never/do not/must not/shall not): ...
don't is absent from this user-visible string too, even though the regex above it detects it. A consumer reading the scan output will believe don't is not scanned — but it is. Together, both instances should be updated in any cleanup commit.
3. grep without -- separator — SUGGESTION (non-exploitable, carried from prior reviews)
Lines 116, 124, 127, 128 call grep without -- before "$f". Not exploitable — all filenames are hardcoded or find-rooted at . (never start with -). Defensive best practice only.
Verdict
No correctness bugs, no security vulnerabilities. The exec-bit fixup (c12ca86) resolved the hygiene gate failure correctly. Three non-blocking findings remain: two are the don't doc/output gap (prior reviews caught one instance; this review adds a second at instruction-surface-scan.sh:162), and one is the defensive grep -- suggestion. All three are safe to merge without fixing. If a cleanup commit is sent, both don't instances should be updated together.
|
Claude finished @kyle-sexton's task in 2m 27s —— View job Security Review — PR #809
ScopeReviewed all 9 changed files:
No GitHub Actions workflow files were changed. FindingsNo CRITICAL or IMPORTANT security vulnerabilities found. Shell script hardening — controls verified independently
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c12ca86413
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…an (#800) The scan audits the user-global CLAUDE.md at a hardcoded $HOME/.claude, but CLAUDE_CONFIG_DIR relocates the entire ~/.claude tree when set (e.g. multi-account setups). In that case the scan read a stale/default file and missed the instructions actually loaded into the session — a failure of the skill's own stated goal. Resolve the user config root from ${CLAUDE_CONFIG_DIR:-$HOME/.claude}. Per https://code.claude.com/docs/en/claude-directory: "every ~/.claude path on this page lives under that directory instead." Test isolation now neutralizes an ambient CLAUDE_CONFIG_DIR on every fixture invocation (HOME alone no longer isolates), plus a positive case seeding a user CLAUDE.md under a distinct config dir. 16/16 checks pass; shellcheck clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fgx6EDAL4cGAFwTNgLpsus
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Claude finished @kyle-sexton's task in 3m 0s —— View job Security Review — PR #809
ScopeThis review covers the full set of 9 changed files and focuses specifically on commit Files with new changes in
All other files are unchanged from prior reviews. FindingsNo CRITICAL or IMPORTANT security vulnerabilities found — including in the new commit. New commit
|
| Control | Status |
|---|---|
set -u in main script; set -uo pipefail in test |
✓ |
NUL-safe surface collection (find -print0 / IFS= read -r -d '') |
✓ |
| All variable expansions quoted | ✓ |
CLAUDE_CONFIG_DIR used only as a path component in [[ -f ]] + grep — no exec or eval |
✓ |
PROHIBITION regex is a hardcoded literal — no user-controlled input reaches -E |
✓ |
printf '%s' (not printf "$var") for candidate_out accumulation |
✓ |
Bounded output: cut -c1-200 per candidate line, head -n 200 on total |
✓ |
2>/dev/null suppresses error disclosure from grep/find |
✓ |
[[ -n "$repo_root" ]] guard before cd "$repo_root" |
✓ |
Test: trap-based cleanup for $FIX; explicit rm -rf for $CFG and $EMPTY |
✓ |
!-injection in SKILL.md emits only integer counts — no file content at load time |
✓ |
| Report-only / human-gated — no unattended file mutation risk | ✓ |
SUGGESTION (non-blocking — carried from prior reviews, confirmed again at HEAD)
grep without -- before filename arguments
instruction-surface-scan.sh:119, :124, :127, :128
Not exploitable: all filenames come from hardcoded entries or find outputs rooted at ., so they never start with -. Adding -- before "$f" on each call is defensive best practice only.
Summary
The CLAUDE_CONFIG_DIR fix in ad3b772 is correct and introduces no new security surface. The implementation is read-only, confined to path construction, and cannot be exploited for command injection or privilege escalation. No CRITICAL or IMPORTANT findings on this PR at any commit. Ready to merge.
|
Claude finished @kyle-sexton's task in 2m 31s —— View job PR Review:
|
|
Post-review classification (babysit worker lane) — findings raised after the prior classification. The two Codex inline findings are answered in-thread. The CLAUDE_CONFIG_DIR one is fixed in
No DEFER-FORBIDDEN class applies to finding 1 (no data-loss, gate, security-posture, or irreversible-action risk — pure output-string completeness). The recurring |
|
Tower blitz (thread-stalled 87 min, checks green): thread 1 resolved — VALID/fixed in |
|
Tower merge-sit: merge-drive lane dark ~5.5h (rate-limit hold since ~10:36Z); tower sitting merges per stall protocol. Gate-verified live: CLEAN, 0 unresolved threads. This was generated by AI (control tower, session 6). |
…s (0.9.0) Remove the audit-model-fit skill (merged concurrently via #809), superseded by audit-instructions. Both answer the same question — locally-owned instruction surfaces vs current model capability — and repo doctrine (one question per skill) admits only one. Grounds for keeping audit-instructions: - fuller catalog: eleven checks (I1-I11) with authority tags and evidence tiers vs audit-model-fit's four (C1-C4, which our I6/I8/I9 + partition cover); - the claude-memory:audit hygiene partition (routes memory-layer hygiene out); - an adversarial fresh-context verify pass over every removal proposal; - the user-locked `audit-instructions` name and the user-approved plan (D1-D7). plugin.json (0.9.0, four-skill description) and the CHANGELOG 0.9.0 Removed note landed in the preceding integration commit during the rebase onto main; main's 0.8.0 audit-model-fit changelog entry is kept as history. Root README catalog regenerated (in sync). Refs #800 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4RMR3G1rHAdNjxTqenVJL
…model-fit (0.9.0) (#870) New `/claude-config:audit-instructions` skill: a read-only audit of the locally-owned Claude Code instruction surfaces — user + project CLAUDE.md, `.claude/rules`, skill bodies + context files, agent definitions, prompt-type hooks, output styles — proposing removals/rewrites of instructions current models no longer need, each finding cited to current official prompting doctrine and gated to the human as proposed diffs. Closes #800 ## What ships - **Skill core** (`plugins/claude-config/skills/audit-instructions/SKILL.md`): Phase A inventory of locally-owned surfaces (plugin cache and managed materializations excluded — findings route to the owning repo), Phase B per-surface subagent lanes with bounded concurrency and a ~20-dispatch cost gate, Phase C fresh-context non-fork adversarial verify pass ("would removing this cause mistakes?"), Phase D report persisted to `${CLAUDE_PLUGIN_DATA}/audit-instructions/last-audit.md` with proposed diffs and a delete-and-watch follow-through section. Report-only; no `--fix`. - **Check catalog** (`reference/criteria.md`, v1.0.0, dated, recheck triggers): eleven checks I1–I11 seeded from official doctrine, each with evidence tier (mechanical/behavioral), authority tag, severity default, per-surface applicability, and source URL. Memory-layer partition: hygiene checks I1–I5 route to `claude-memory:audit` on CLAUDE.md/rules surfaces (that plugin's C1–C8 own the hygiene layer); model-era checks I6–I11 apply everywhere. - **Deterministic spine** (`scripts/instruction-scan.sh` + test): advisory grep pre-scan seeding I6 (bare prohibitions without rationale) and I10 (reasoning-echo directives) candidates; always exit 0, exit 2 on missing dep, `--count`; no jq. - **Evals** (`evals/evals.json`, 5 cases): read-only bare invocation, scope-boundary route-out, plugin-cache routing, example-block nuance, verify-pass demotion. - **Integration**: claude-config → 0.9.0, CHANGELOG, README row + subsection, generated root catalog. - **Supersedes `audit-model-fit`** (merged concurrently in #809 from the same underlying issue #800, racing this branch's live work-item lease): both skills answered the same question — instruction surfaces vs current model capability — and the repo's one-question-per-skill doctrine forbids shipping both. `audit-instructions` survives on grounds recorded in the CHANGELOG `### Removed` entry: the user-locked name from the naming pipeline (the "constraints/unhobbling" framing was explicitly rejected there), the user-approved plan (D1–D7) versus automated triage only, and a strictly-covering catalog (their C1–C4 ≈ I6/I8/I9 plus the claude-memory hygiene partition, authority tags, evidence tiers, and the adversarial verify pass). Absorb candidates from audit-model-fit (pre-computed inventory injection, a dedicated step-list-culling eval) are noted below as follow-up material. - **Fix folded in**: the stale `health`-skill route-out reference (claude-memory's skill is named `audit`) corrected across audit, audit-automation-gaps, audit-permission-grants, and the plugin README; setup skill prerequisite notes reconciled for the fourth (grep-only) skill. ## Verification - Local gates green: skill-quality check-skill (0 errors), changed-skills regression gate, validate-plugin-contracts, portability lint, markdownlint (11 files), changelog parity, generate-catalog `--check`, scanner test 22/22, evals schema validation, full plugin test suite. - Each implementation phase was verified by a fresh-context acceptance verifier against binary criteria (9/9, 5/5, 8/8 PASS), plus an independent fresh-context branch review (4 IMPORTANT findings, all fixed and re-confirmed). ## Related - #809 — the concurrent `audit-model-fit` PR this change supersedes (its 0.8.0 CHANGELOG entry is preserved as history; absorb candidates — `!`-injected surface inventory, step-list-culling eval — are follow-up material). - #798 — sibling `library_dir` indirection effort (separate item, not folded in). - #868 — follow-up: reciprocal boundary note in `claude-memory:audit` (separate-plugin change split out of this PR). - melodic-software/knowledge-corpus#3 — digest whose context-trimming findings seeded the check catalog. <details> <summary>Approved PLAN.md (topic slice pruned pre-merge; snapshot as executed)</summary> # claude-config-audit-instructions ## Brief ### TLDR - New skill `/claude-config:audit-instructions` (issue #800): read-only audit sweeping the locally-owned Claude Code instruction surfaces — user + project CLAUDE.md, skill bodies + context files, agent definitions, `.claude/rules/**/*.md`, prompt-type hooks, output styles — proposing removals/rewrites of instructions current models no longer need. - Check catalog seeded from the 11 sourced seeds in knowledge-corpus `.work/youtube-watch/how-i-plan-build-and-run-loops-with-clau-aVO6E181cNU/research/findings/context-trimming.md`, each check cited to its official source. - Findings tiered by evidence class: **mechanical** (pattern-detectable: bare prohibitions without rationale, inferable/redundant content, misplaced only-sometimes-relevant content, rule-to-hook candidates, show-your-thinking instructions, non-steering example blocks) vs **behavioral** (pruning-bar counterfactuals — "would removing this cause mistakes?" — whose ground truth is observed behavior, not static analysis). - Execution shape: per-surface subagent lanes sharing one check catalog; large lanes (skills) fan out further. Every removal proposal passes an adversarial "would removing this cause mistakes?" verify pass before reaching the human-gated diff. - Output: findings report + proposed diffs, human-gated, never auto-applied. Upstream-owned findings route out (standards-managed materializations → melodic-software/standards per sync-manifest; marketplace plugin content → claude-code-plugins issues), never edited in place. ### Goal A claude-config audit-family skill that, on demand or after a model-generation upgrade, tells the operator exactly which instruction lines across their locally-owned Claude Code configuration are no longer earning their context cost — with each finding cited to current official prompting doctrine, classified by how confident the evidence can be, and packaged as a proposed diff the human approves or rejects — so instruction surfaces shrink as models improve instead of accreting prior-model-era scar tissue. ### Constraints - Name locked: `audit-instructions` (this session, via naming pipeline). Family grammar: `audit` = read-only findings report; mutation never on bare invocation (`docs/PLUGIN-PHILOSOPHY.md` naming table). - Human-gated output only: findings report + proposed diffs; never auto-applies. No `--fix` in scope for this issue. - Scope bound (locked in issue #800 follow-up): audit only locally-owned surfaces (user CLAUDE.md, `~/.claude/rules`, project `.claude/`). Marketplace plugin cache content is upstream-owned — findings there route to claude-code-plugins issues, never local cache edits. - Cross-repo routing: findings inside standards-managed materializations route upstream to melodic-software/standards per `standards/distribution/sync-manifest.yml` ownership. - Composes by pointer, distinct intents (locked): `skill-quality:check` (structural lint), `docs-hygiene:compress` (token brevity), `claude-config:audit` (config-file mechanics — settings.json, .mcp.json, hooks wiring). This skill owns instruction *content* vs current model capability; no overlap restated. - Check catalog is cited doctrine, not copied prose: each check carries its source URL (code.claude.com best-practices, platform.claude.com Fable 5 + prompting best-practices pages) with verified date and a recheck trigger — model-specific pages get superseded per release. - Repo obligations: plugin version bump + CHANGELOG entry, skill-quality:check pass, validate-plugin-contracts gate, commit via `git commit -F - --cleanup=verbatim`. ### Acceptance criteria 1. `plugins/claude-config/skills/audit-instructions/` ships; frontmatter description carries the discovery triggers (post-model-upgrade, "prune my CLAUDE.md", "are my instructions holding the model back") and the distinguishing object in its first clause. 2. Check catalog covers all 11 seeds (pinned in the plan appendix); every check cites its official source and declares its surface applicability per the claude-memory partition (hygiene checks I1–I5 route to claude-memory:audit on memory-layer surfaces when that plugin is installed). 3. Findings report tiers every finding mechanical vs behavioral; every removal/rewrite proposal carries the adversarial verify verdict before it is surfaced. 4. Bare invocation is read-only end-to-end; diffs are proposed artifacts, never applied. 5. Scope guard enforced in skill flow: plugin-cache and standards-managed paths are excluded from the editable set and their findings emitted as routing recommendations instead. 6. Per-surface lane execution shape (subagent per surface, shared catalog, fan-out for skills) is the documented default flow. 7. skill-quality:check passes for the new skill; claude-config version bumped with CHANGELOG entry; PR closes #800. ### Captured assumptions - The 11-seed catalog reflects official doctrine as of 2026-07-21, re-verified against live docs this session (research pass) — revisit on next frontier model release or when either prompting page changes (recheck trigger recorded in the skill's sources). - Attribution (research-verified 2026-07-21): the pruning bar "Would removing this cause Claude to make mistakes?" and the delete-and-watch loop ("test changes by observing whether Claude's behavior actually shifts") are Anthropic best-practices doctrine. Boris Cherny's documented practice is the additive write-it-down loop plus `/checkup` (dedup, split big CLAUDE.md into nested files + skills); a periodic full-delete ritual is unconfirmed in any primary source — the skill must not cite it as his. - Bare-prohibition remediation (research-verified): the docs' primary target form is positive reframing ("tell Claude what to do instead of what not to do"); adding rationale is separately supported ("give the reason, not only the request"). The check offers positive rewrite first, prohibition-plus-rationale as the fallback where a genuine hard "never" survives. - Examples remain officially recommended (3–5, format/tone/structure steering) for all current models including Fable-class — the audit flags example blocks only when they are behavioral scaffolding pinning the model's approach, never format steering; revisit if the best-practices page drops the recommendation. - Behavioral-tier caution is itself sourced: Anthropic's postmortem on the ~80% system-prompt cut (InfoQ/VentureBeat coverage) records that narrow evals missed a ~3% regression — grounding why behavioral findings ship as proposals with the delete-and-watch loop, never as confident removals. ### Out-of-scope - #798 (`library_dir` indirection) — separate item, not folded in. - Any auto-apply / autofix mode. - Building an eval harness for instruction A/B testing (the report may *recommend* the empirical loop; shipping tooling for it is not this issue). - Auditing upstream plugin content in place (routing-only, per scope bound). ### Deferred questions - Empirical validation posture: does the findings report prescribe the delete-and-watch + re-add-on-mistake loop (and cheap A/B for example blocks) as its recommended follow-through, or stop at the human-gated diff? — defer until /planning:plan; **arbiter: USER-RESERVED** (changes report shape and acceptance criterion 3's follow-through). - Upgrade-trigger mechanics: how "upgrade-triggered" fires (manual invocation documented as the trigger vs any automation/hook seam) — defer until /planning:plan; **arbiter: /architect** (issue text already sanctions "upgrade-triggered or on-demand"; execution-shape decision). ## Plan Standards grounding: repo `CLAUDE.md` (fresh-docs mandate, design rules), `docs/PLUGIN-PHILOSOPHY.md` (naming, config ownership, fresh-eyes checkpoints, two-lane convention posture, prerequisites/degrade, cross-platform), skill-quality 18-check gate, `scripts/validate-plugin-contracts.mjs`, portability-lint, changelog-parity. Precedent map: memory slice `EXPLORE.md` (canonical checkout, `.work/claude-config-audit-instructions/`). ### Open Decisions (resolved pre-plan; override any at the gate) - **D1 claude-memory boundary — complementary partition** (revised after plan review verified `claude-memory:audit` C1–C8 already implement the hygiene seeds for CLAUDE.md/rules: C2 = deletion test, C1 = line budget, C3 = placement, C5 = inferable content, C8 = enforcement hierarchy). Partition, not overlap-with-disclaimer: - Memory-layer surfaces (CLAUDE.md, CLAUDE.local.md, `.claude/rules`): this skill runs ONLY the model-era checks (I6–I11); hygiene checks route out to `claude-memory:audit` (gated "when installed", fallback = one-line pointer to the official include/exclude table). - Non-memory surfaces (skill bodies + context files, agent definitions, prompt-type hooks, output styles): FULL catalog applies — no incumbent auditor (`skill-quality:check` is structural lint only). - Trigger disambiguation: description triggers are capability-framed ('after a model upgrade', 'too prescriptive', 'instructions the model no longer needs', 'holding the model back') — NOT 'prune my CLAUDE.md'/'audit CLAUDE.md' ('prune instructions' family is claude-memory:audit's claimed trigger space). - Reconciliation: fix the stale route-out reference in sibling `audit-permission-grants/SKILL.md` (names a nonexistent claude-memory `health` skill; actual name `audit`) — same plugin, this PR (Phase 5). A reciprocal boundary note in claude-memory:audit itself is a separate-plugin change → follow-up issue filed at PR time (cited in `## Related`). - **D2 catalog carrier** — `reference/criteria.md`, versioned + dated, per-check authority tag (mcp-tools idiom) — 11 checks don't warrant machine-health JSONC machinery. Escape hatch recorded: graduate to `catalog/checks.jsonc` + schema if the catalog grows past ~20 checks or gains self-maintenance counters. - **D3 severity vocabulary** — `error`/`warning`/`info` severity axis (family precedent) + independent **authority** tag (`ANTHROPIC-DOCS` / `TALK` / `OPINION`) + independent **evidence tier** (`mechanical` / `behavioral`). Three orthogonal axes, no conflation. - **D4 deterministic spine in v1** — YES, minimal: one advisory scanner script (bare-prohibition lines and reasoning-echo phrases), always exit 0, `exit 2` on missing dep, `--count` flag, co-located test — claude-memory deterministic-spine + permission-grants script idiom. It seeds the mechanical tier; the model layer refines its hits (a grep cannot judge rationale presence reliably — the script marks candidates, the lane classifies). - **D5 plugin version** — 0.8.0 (additive skill = minor; changelog-parity gate satisfied by a new `## [0.8.0]` entry). - **D6 empirical-validation posture (was USER-RESERVED)** — the report ENDS with a "Recommended follow-through" section prescribing the official behavioral loop (delete-and-watch: "test changes by observing whether Claude's behavior actually shifts"; re-add-on-mistake as the compounding safety net; cheap A/B against the no-example default for example blocks). Prose guidance only — no eval tooling shipped (Brief out-of-scope). Ratified by approving this plan. - **D7 upgrade-trigger mechanics** — manual invocation only in v1; the description's trigger phrases carry the post-upgrade moment; no hook/automation (native-first: no lifecycle event exists for "model changed"; a consumer can pair with /loop or a scheduled routine — documented as a one-line note, not built). ### Phase 1: Skill core — SKILL.md [DONE] Create `plugins/claude-config/skills/audit-instructions/SKILL.md`: - Frontmatter: `name: audit-instructions`; `description` with distinguishing object in first clause + single-quoted 'Use when' triggers per D1's capability-framed set ('after a model upgrade', 'are my instructions holding the model back', 'instructions the model no longer needs', 'audit instructions', 'instruction audit', 'too prescriptive') + trailing "Report-only." note; `argument-hint: "[scope] — scope: claude-md|rules|skills|agents|hooks|output-styles|all (default: all)"`; `user-invocable: true`; `disable-model-invocation: false`. Description+when_to_use ≤1536 chars (check 2). - Body sections (permission-grants lean idiom, <200 lines soft target): Purpose; **Scope boundary (route out)** per D1 — every cross-plugin route seam-phrased per `docs/conventions/seam-phrasing/README.md` (gate "when <plugin> is installed" + stated fallback); Arguments; **Phase A Inventory** — enumerate locally-owned surfaces: user CLAUDE.md, user rules dir, user skills (`~/.claude/skills`) and agents, project CLAUDE.md, project `.claude/` (rules, skills, agents, output styles), prompt-type hooks — exact user-level paths (`~/.claude/rules` et al.) doc-verified at implementation per the fresh-docs mandate; EXCLUDE plugin cache (upstream-owned → findings become routing recommendations to the owning repo's tracker) and managed materializations (detection-first, org-agnostic: when the consuming repo's own docs declare a managed/locally-owned distribution seam — any sync-manifest convention — route managed-file findings upstream; no such declaration → no exclusion; no publisher names or hardcoded manifest paths in the skill); **Phase B Per-surface lanes** — fresh read-only subagent per surface sharing `reference/criteria.md` with D1's per-surface check partition, bounded concurrency 3–5, skills lane fans out per skill, cost gate: confirm with user before total dispatches (lanes + verifiers) exceed ~20, tier-transparency line naming surfaces run AND skipped; **Phase C Verify pass** — removal/rewrite proposals re-judged by fresh-context NON-FORK subagents prompted to refute ("would removing this cause mistakes? argue the instruction is still load-bearing") — self-grade class, fresh-eyes mandatory; batched one verifier per surface (not per finding), counted under the same ~20 dispatch gate; refuted proposals demoted to `info`/kept; **Phase D Report** — findings table `| # | Check | Surface:Line | Severity | Tier | Authority | Finding | Proposed change |`, proposed diffs as fenced blocks per finding, "a clean audit is a valid outcome", "Recommended follow-through" per D6, routing subsection for out-of-scope surfaces; report persisted to `${CLAUDE_PLUGIN_DATA}/audit-instructions/last-audit.md` (claude-memory persistence idiom) + chat summary; Gotchas (inline `## Gotchas` — check 11); What this skill does NOT do (never edits, never auto-files, not a brevity pass, not memory-layer hygiene). - Read-only wording: "This skill is report-only. There is no `--fix`: instruction files are the operator's voice — every change is applied by the human (or explicitly delegated afterwards), never by this skill." **Sanity Check** (runs at the Phase 1–3 commit boundary — check 5 needs criteria.md and the script to exist before check-skill passes): `CHECK_SKILL_SKILLS_ROOT="$PWD/plugins/claude-config/skills" bash plugins/skill-quality/scripts/check-skill.sh audit-instructions` exits 0 with no FAIL; `grep -c "Scope boundary" plugins/claude-config/skills/audit-instructions/SKILL.md` ≥ 1; `grep -Ec "^## Gotchas" SKILL.md` = 1. ### Phase 2: Check catalog — reference/criteria.md [DONE] Create `plugins/claude-config/skills/audit-instructions/reference/criteria.md`: - Header: `Version: 1.0.0`, `Last updated: 2026-07-21`, recheck triggers (next frontier model release; change to either prompting page; best-practices page change), source URL list. - 11 checks `I1`–`I11` mapping the seeds: I1 line-necessity bar; I2 length/skimmability; I3 broad-applicability placement (CLAUDE.md vs skill); I4 inferable/redundant content (include/exclude table); I5 rule-to-hook conversion; I6 bare prohibition → positive reframing first, prohibition-plus-rationale fallback for genuine hard nevers; I7 reason-with-request; I8 model-era re-audit (prior-model workarounds, too-prescriptive skills); I9 example hygiene (keep 3–5 format/tone steering; flag behavioral scaffolding); I10 reasoning-echo instructions (`reasoning_extraction` hazard); I11 CLI-over-MCP. - Each check: id, title, evidence tier (mechanical/behavioral), authority tag, severity default, **surface applicability per D1's partition** (I1–I5: non-memory surfaces only, memory-layer occurrences route to claude-memory:audit when installed; I6–I11: all surfaces), detection guidance, remediation form, source URL. Point-don't-copy: quote only the decisive line per source. **Sanity Check:** `grep -Ec "^### I(10|11|[1-9]):" reference/criteria.md` = 11; `grep -c "https://" reference/criteria.md` ≥ 5; check-skill.sh check 5 resolves the SKILL.md → criteria.md reference (exit 0 overall). ### Phase 3: Deterministic spine — scripts [DONE] Create `plugins/claude-config/skills/audit-instructions/scripts/instruction-scan.sh` + `…/scripts/instruction-scan.test.sh` (skill-local, sibling idiom — NOT repo-root `scripts/`) (per D4): scans given file paths for I6 candidates (bare `never|do not|don't` lines lacking because/since/so-that rationale markers) and I10 candidates (show-your-thinking phrases); outputs `file:line:check-id` advisory rows; `--count` flag; always exit 0 (advisory), exit 2 if a required tool is missing; POSIX-lean bash, no jq requirement; Windows path via Git Bash documented in SKILL.md prerequisites. Add the script to `skills/setup/SKILL.md` tool inventory. **Sanity Check:** `bash plugins/claude-config/skills/audit-instructions/scripts/instruction-scan.test.sh` exits 0; `bash …/instruction-scan.sh --count <fixture>` prints an integer and exits 0. ### Phase 4: Evals [DONE] Create `evals/evals.json` (schema `plugins/skill-quality/reference/evals.schema.json`): cases — (1) bare invocation stays read-only (expectations: no Edit/Write of audited files, report produced); (2) scope-boundary-routes-out (memory-health request → points at claude-memory); (3) plugin-cache finding routed not edited; (4) example-block nuance (format-steering examples NOT flagged); (5) verify-pass demotion (refuted removal not presented as error). **Sanity Check:** `pipx run check-jsonschema --schemafile plugins/skill-quality/reference/evals.schema.json plugins/claude-config/skills/audit-instructions/evals/evals.json` exits 0 — the Python tool CI uses (`check-jsonschema` is NOT an npm package; prior session verified npx form fails). ### Phase 5: Plugin integration [DONE] - `plugins/claude-config/.claude-plugin/plugin.json`: version → 0.8.0; description string extended to four skills. - `plugins/claude-config/CHANGELOG.md`: new `## [0.8.0]` → `### Added` entry (+ `### Fixed` for the stale route-out reference below). - `plugins/claude-config/README.md`: one-question table row + "What each skill does" subsection. - `plugins/claude-config/skills/audit-permission-grants/SKILL.md`: fix stale route-out — "`health` skill in the `claude-memory` plugin" → the actual `audit` skill (D1 reconciliation, same plugin). - Root README catalog is GENERATED — run `node scripts/generate-catalog.mjs` after the plugin.json description change (hand-editing drifts; CI runs `--check`). **Sanity Check:** `bash scripts/check-changelog-parity.sh --check-bump origin/main` exits 0; `grep -c "audit-instructions" plugins/claude-config/README.md` ≥ 2; `node scripts/validate-plugin-contracts.mjs` exits 0; `node scripts/generate-catalog.mjs --check` exits 0; `grep -c "health" plugins/claude-config/skills/audit-permission-grants/SKILL.md` returns 0 for the stale-reference form. ### Phase 6: Gates, review, PR [DOING] Full local gate run (check-skill, contracts, portability, markdownlint, changelog parity, plugin tests), independent review pass (fresh-context reviewer per repo doctrine), then PR: title Conventional Commits; body `Closes #800`, non-empty `## Related` (#798 sibling effort, knowledge-corpus#3 digest provenance), approved PLAN.md in `<details>`; prune topic slice in final pre-merge commit. Merge surfaced to user per never-merge boundary. **Sanity Check:** all listed gate commands exit 0 locally; `gh pr view --json body` contains `Closes #800` and a `## Related` section; ci-status green. ### Appendix: pinned seed catalog (durable copy — source repo's `.work/` slice is ephemeral) Source: knowledge-corpus, memory slice `.work/youtube-watch/how-i-plan-build-and-run-loops-with-clau-aVO6E181cNU/research/findings/context-trimming.md` (committed in knowledge-corpus#3). Pinned 2026-07-21 so AC2 stays verifiable if that slice moves: | Seed | Check | Summary | Source | |---|---|---|---| | 1 | I1 | Line-necessity: "Would removing this cause Claude to make mistakes?" else cut | code.claude.com/docs/en/best-practices | | 2 | I2 | Length/skimmability gate; bloat causes instruction-ignoring | code.claude.com/docs/en/best-practices | | 3 | I3 | Only-sometimes-relevant content → skill, not always-loaded CLAUDE.md | code.claude.com/docs/en/best-practices | | 4 | I4 | Exclude inferable content (code, standard conventions, linked API docs, truisms) | code.claude.com/docs/en/best-practices (include/exclude table) | | 5 | I5 | Rule already followed or 100%-required → hook or delete | code.claude.com/docs/en/best-practices | | 6 | I6 | Bare prohibition → positive reframing first; rationale fallback for hard nevers | platform.claude.com …/claude-prompting-best-practices | | 7 | I7 | Carry intent/motivation with instructions | same + …/prompting-claude-fable-5 ("Give the reason") | | 8 | I8 | Release-time re-audit; prior-model skills often too prescriptive, degrade quality | platform.claude.com …/prompting-claude-fable-5 | | 9 | I9 | Examples only for format/tone/structure steering (3–5); flag behavioral scaffolding | …/claude-prompting-best-practices + Fable 5 page | | 10 | I10 | Flag show-your-thinking/reasoning-echo instructions (`reasoning_extraction` refusal) | platform.claude.com …/prompting-claude-fable-5 | | 11 | I11 | Prefer CLI over MCP where equivalent (context cost) | code.claude.com/docs/en/best-practices | ## Blast radius LOW — additive markdown skill + one advisory script inside one plugin; no runtime code paths of other skills touched; plugin.json description/version edits are metadata. Cross-plugin surface limited to README/catalog rows. No consumer-breaking contract changes. ## Stress-test summary Fresh-context plan reviewer (Step 3) returned 1 CRITICAL / 6 IMPORTANT / 4 SUGGESTION; all verified against the repo before applying. Applied: D1 rewritten as a complementary partition (claude-memory:audit C1–C8 verified to already cover the hygiene seeds on memory-layer surfaces), capability-framed trigger set, seam-phrased route-outs, stale `health` reference fix folded into Phase 5, `pipx run check-jsonschema` (npm package does not exist), generated-catalog regen + `--check` sanity, Phase 1–3 single commit unit (check 5 reference resolution), skill-local script path, pinned seed appendix, org-agnostic managed-seam detection, verify-pass batching under the shared ~20-dispatch gate, surface enumeration completed (user skills/agents) with user-path doc-verification note, report persistence to `${CLAUDE_PLUGIN_DATA}`. Formal /devils-advocate skipped: blast radius LOW, no triggers matched. ## Execution shape Sequential, single scope-fenced implementation worker in this worktree (orchestrator never edits source; work-items:work dispatch posture). Phases 1–3 are ONE commit unit (skill-quality check 5 requires SKILL.md's criteria.md and script references to resolve), then 4, 5 as separate commits; 6 closes. No parallel wave: file sets are small and Phase 5 depends on 1–4 outputs. | Phase | Surface | Basis | |---|---|---| | 1–5 | implementation worker subagent (this worktree) | mechanical authoring against locked plan; orchestrator-never-edits | | 6 | orchestrator (gates, PR) + fresh-context reviewer | judgment + repo doctrine (producer ≠ critic) | Sequential fallback: n/a (already sequential). ## Open questions None unresolved — D1–D7 locked above pending gate approval; D6 was USER-RESERVED and is ratified by plan approval. ## Handoff to implementation ### User-approval gates - Plan approval (this gate) ratifies D6 (follow-through section) and D1 (stay in claude-config). - PR merge is surfaced to the user (never-merge boundary) unless the user delegates to the babysit worker tier. ### Execution shape ([EXEC-SHAPE] tagged) - [EXEC-SHAPE] Single worker, sequential phases, commit boundaries as listed in Execution shape. - Worker ALLOWED: `plugins/claude-config/**` (includes the audit-permission-grants stale-ref fix), root `README.md` ONLY via `node scripts/generate-catalog.mjs` (generated, never hand-edited). FORBIDDEN: other plugins' files except read; `.github/**`; `docs/topics/**` (PLAN.md status tags are orchestrator-only). ### Mechanical work Commit per phase boundary with `-F - --cleanup=verbatim`; run Phase sanity checks at each green checkpoint; divergence from plan → back to /planning:plan review, never push through. </details> 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: melodic-standards-sync[bot] <300666570+melodic-standards-sync[bot]@users.noreply.github.com>
…tions (0.9.1) (#892) Adds the step-list-culling eval case to `claude-config:audit-instructions` — the second of two absorb candidates recorded in #870 when `audit-model-fit` was superseded. Closes #889 ## What ships - **New eval case `step-list-culled-not-preserved`** (`evals/evals.json`, id 6): a mechanical 9-step numbered procedure must be flagged as an I8 (Model-era re-audit) over-prescriptive finding and culled to intent plus hard constraints — steps encoding genuine ordering, safety gates, or external contracts kept — never preserved verbatim. Modeled on the superseded suite's C2 analog, re-phrased against the I8 catalog. - **claude-config 0.9.1** + CHANGELOG entry per changelog-parity. ## Verification - `check-jsonschema` against `plugins/skill-quality/reference/evals.schema.json`: ok. - Local gates green: check-changed-skills (check-skill PASS, 0 errors), changelog-parity `--check` + `--check-bump`, generate-catalog `--check`, validate-plugin-contracts, markdownlint. - Fresh-context acceptance verifier: 5/5 criteria PASS (case shape/uniqueness, I8 mapping confirmed against `reference/criteria.md`, no mojibake from the historical source, bump + CHANGELOG parity, commit subject). ## Related - #870 — supersede PR that recorded this absorb candidate as follow-up material. - #888 — sibling absorb candidate (`!`-injected surface inventory), tracked separately. - #800 / #809 — originating issue and the superseded `audit-model-fit` PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Summary
Adds
audit-model-fit, a fourth audit skill to theclaude-configplugin. It sweeps the local ClaudeCode instruction surfaces — user + project
CLAUDE.md, skillSKILL.mdbodies + context files, agentdefinitions,
.claude/rules/**, prompt-type hooks and output styles — for deterministic constraintsthat hobble newer, more capable models, and proposes removals/rewrites. It is report-only and
human-gated: it presents findings plus proposed diffs and never edits an instruction file itself.
Motivation (per the issue): instructions written to make a prior, weaker model behave become carrying
cost and can degrade a capable model's output. Source guidance — Anthropic's Fable 5 prompting doc
("re-evaluate which instructions, tools, and guardrails are still needed") and the Claude Code
best-practices pruning bar ("would removing this cause Claude to make mistakes?").
Implementation
New skill directory
plugins/claude-config/skills/audit-model-fit/, matching the siblingaudit-*skills' shape (frontmatter, phased
SKILL.md,reference/criteria,scripts/+.test.sh,evals/evals.json):SKILL.md— pre-computed!-injected surface inventory, purpose, scope boundary (compose-with),cross-repo routing, scope-filter arguments, and three phases (enumerate → classify → report +
proposed diffs). Report-only; no
--fix.reference/criteria.md— the check catalog seeded by the issue: C1 bare prohibitions(rewrite to add rationale, not delete), C2 over-prescriptive step lists (cull to intent + hard
constraints), C3 over-constraining example blocks (trim toward 3–5, not a blanket ban), C4
stale model-era workarounds — all measured against the pruning bar. Cites the issue's source URLs.
scripts/instruction-surface-scan.sh(+.test.sh) — deterministic, advisory (always exits 0).Enumerates the surfaces and flags the two grep-able smells (C1 bare prohibitions, C3 example-dense
files) as candidates; the judgment (C2, C4, the pruning bar, rationale-presence) stays in the skill
body. Prose surfaces are grep-scanned; hook scripts are counted-only (their prompt text is embedded
in code) and read by hand in Phase 2. Modes: default inventory,
--candidates,--count.evals/evals.json— 9 rich-form cases covering the subtle distinctions: rewrite-not-delete,examples-not-banned, step-list culling, stale-workaround removal, standards-managed upstream routing,
never-auto-applies, route-out to siblings, and clean-sweep-is-valid.
Composition (pointer-only, distinct intents):
claude-memory:audit(instruction-layer health onthe same surfaces — route health/structure findings there),
skill-quality:check(structure),docs-hygiene:compress(token brevity),claude-config:audit(config-file correctness).Cross-repo routing: findings inside
melodic-software/standards-managed materializations routeupstream per that repo's
distribution/sync-manifest.ymlrather than being edited in place (documentedin the skill even though the manifest is not vendored into this repo).
Plus the plugin's canonical shared updates:
plugin.jsonversion0.7.1→0.8.0and description"Three" → "Four audit skills"; matching
README.md(table row + section) andCHANGELOG.md[0.8.0]entry.
An independent fresh-context reviewer (withheld rationale) audited the skill; its IMPORTANT findings
were folded in before commit — the missing
claude-memory:auditroute, a dead helper function removed,C2/C4 eval coverage added, the hooks counted-only distinction documented, and a C3 threshold-boundary
test.
Verification
All commands run locally in this worktree.
Skill-quality static gate (
skill-quality:check, skills-root pointed at the plugin):The single WARN (no
## Gotchassurface) matches the siblingaudit-permission-grantsexactly — nosibling in the plugin ships a Gotchas surface, and this is a brand-new skill with no failure history.
Script tests (
instruction-surface-scan.test.sh):All 14 checks passed.— covers surfaceenumeration, both grep-able smells,
--count/--candidatesmodes, the C3 threshold boundary (a5-block file is not flagged), and empty-repo safety.
Shellcheck: clean on both scripts.
Evals schema:
check-jsonschemaagainstskill-quality/reference/evals.schema.json→ok.Markdownlint (
markdownlint-cli2) onSKILL.md,criteria.md,README.md,CHANGELOG.md:Summary: 0 error(s).JSON validity:
plugin.jsonandevals.jsonvalid.Commit is signed (ED25519, verified) and created via
git commit -F -.Closes #800
Related
claude-config: add model-upgrade unhobbling audit).claude-loop-practices(2026-07-21), per the issue body:home = claude-config audit family, output = findings + proposed diffs (human-gated), cross-repo
routing upstream for standards-managed materializations, composition intents.
audit-*skills whose structure this follows:audit,audit-automation-gaps,audit-permission-grants.🤖 Generated with Claude Code
https://claude.ai/code/session_01K1V3gkrfSf75isB8MiDy3o