test(claude-config): add step-list-culling eval case to audit-instructions (0.9.1)

[kyle-sexton]

Adds the step-list-culling eval case to claude-config:audit-instructions — the second of two absorb candidates recorded in #870 when audit-model-fit was superseded.

Closes #889

What ships

• New eval case step-list-culled-not-preserved (evals/evals.json, id 6): a mechanical 9-step numbered procedure must be flagged as an I8 (Model-era re-audit) over-prescriptive finding and culled to intent plus hard constraints — steps encoding genuine ordering, safety gates, or external contracts kept — never preserved verbatim. Modeled on the superseded suite's C2 analog, re-phrased against the I8 catalog.
• claude-config 0.9.1 + CHANGELOG entry per changelog-parity.

Verification

• check-jsonschema against plugins/skill-quality/reference/evals.schema.json: ok.
• Local gates green: check-changed-skills (check-skill PASS, 0 errors), changelog-parity --check + --check-bump, generate-catalog --check, validate-plugin-contracts, markdownlint.
• Fresh-context acceptance verifier: 5/5 criteria PASS (case shape/uniqueness, I8 mapping confirmed against reference/criteria.md, no mojibake from the historical source, bump + CHANGELOG parity, commit subject).

Related

• feat(claude-config): add audit-instructions skill, superseding audit-model-fit (0.9.0) #870 — supersede PR that recorded this absorb candidate as follow-up material.
• claude-config: audit-instructions — evaluate pre-computed !-injected surface inventory (absorb candidate from audit-model-fit) #888 — sibling absorb candidate (!-injected surface inventory), tracked separately.
• claude-config: add model-upgrade unhobbling audit — sweep instruction surfaces for constraints newer models no longer need #800 / feat(claude-config): add audit-model-fit unhobbling audit #809 — originating issue and the superseded audit-model-fit PR.

🤖 Generated with Claude Code

[claude]

Claude finished @kyle-sexton's task in 1m 26s —— View job

---

PR Review

• Gather context
• Read changed files
• Review for correctness, security, and guideline alignment
• Post findings

---

Summary

Adds a single eval case (id: 6, step-list-culled-not-preserved) to audit-instructions, bumps the plugin to 0.9.1, and records a CHANGELOG entry. All three files are in scope; nothing else changed.

---

evals/evals.json — new case id 6

Schema compliance. Validated against plugins/skill-quality/reference/evals.schema.json:

• id: 6 (integer) — unique; no collision with ids 1–5. ✓
• name: "step-list-culled-not-preserved" — matches ^[a-z0-9]+(-[a-z0-9]+)*$. ✓
• prompt, expected_output, files: [], expectations — all present and well-typed. ✓

I8 mapping. reference/criteria.md I8 reads: "over-prescriptive step lists — instructions enumerating behaviors a current model handles from a brief instruction." A 9-step git-PR workflow is a textbook example of that pattern. The mapping is correct. ✓

Expectation coverage. The three expectations exercise three orthogonal obligations:

#	What it tests
1	Classification — skill identifies the finding as I8
2	Remediation direction — culling to intent + hard constraints, not verbatim preservation
3	Preservation nuance — genuine ordering / safety-gates / external-contract steps are kept

This mirrors the structure of the other five cases (each covers detect, remediate, and a nuance). ✓

One minor observation: expected_output is a compound sentence that asserts both the positive (culls to intent) and negative (does not preserve all 9 verbatim) in a single blob. This is consistent with how other cases in the file are written (e.g. id 4 and id 5 do the same), so it's not a defect — just an observation that the expectations bullets are the durable, machine-checkable layer here.

---

plugin.json — version bump 0.9.0 → 0.9.1

Patch increment for a non-breaking eval-only addition. Correct per semver. ✓

---

CHANGELOG.md — [0.9.1] entry

Follows Keep a Changelog ### Added convention, matches the existing [0.9.0] entry's style (no explicit date, which is consistent with prior entries), and accurately describes the absorbed case and its lineage. ✓

---

No issues found

The change is minimal, schema-valid, correctly mapped to I8, and accompanied by the required housekeeping (version bump + CHANGELOG). No security concerns, no structural problems, no missing tests for the new behavior. The commit subject follows Conventional Commits with the test(claude-config): scope.

Verdict: ready to merge.

[claude]

Claude finished @kyle-sexton's task in 50s —— View job

---

Security Review — PR #892

• Gather context and understand request
• Fetch PR diff
• Read changed files
• Perform security analysis
• Post findings

---

Findings

No security issues found.

The three changed files are exclusively static, non-executable data:

File	Change	Security relevance
plugin.json	Version string bump 0.9.0 → 0.9.1	No risk — purely declarative metadata
CHANGELOG.md	Prose documentation entry	No risk — documentation only
evals/evals.json	New JSON eval case (id 6)	No risk — see analysis below

evals.json eval case analysis (id 6):

• prompt — natural-language scenario description; no dynamic interpolation, no shell escaping, no template expansion. Treated as opaque text by the eval harness.
• expected_output — static string; describes expected LLM behavior, not executable.
• expectations — static string array; no code execution paths.
• files — empty array []; no file-path references that could introduce path traversal.

Checked for and found nothing across:

• Injection surfaces (command, SQL, path, template) — none present; all values are inert strings consumed by an LLM, not a shell or query executor.
• Secret / credential exposure — no tokens, API keys, or passwords.
• GitHub Actions patterns (pull_request_target, workflow_run, run: with github context, permission widening) — no workflow files changed.
• Dependency / supply-chain risk (loosened or unpinned pins) — no dependency manifests changed.
• Access-control or settings drift — no settings.json, MCP configs, or permission files touched.

This is a patch-level data-only addition with no security surface.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ccdc14e402

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[claude]

Claude finished @kyle-sexton's task in 1m 42s —— View job

---

PR Review

• Gather context
• Read changed files
• Assess Codex feedback on verify-pass expectation
• Review for correctness, security, and guideline alignment
• Post findings

---

Summary

Two commits: the first adds the step-list-culled-not-preserved eval case (id 6); the second directly addresses the Codex inline comment by pinning the I8 behavioral-tier verify-pass requirement. All three changed files are in scope.

---

evals/evals.json — new case id 6

Codex feedback status: resolved in commit 7db35fc. The comment asked for an expectation that the cull be surfaced as a verified behavioral proposal. That expectation now exists as expectation 4, and the same language appears in expected_output:

"Because I8 is behavioral-tier, the cull reaches the report as a verify-pass-vetted behavioral proposal, not an unverified confident removal."

This directly satisfies the Codex concern. ✓

I8 mapping. reference/criteria.md I8 reads: "prior-model workarounds and over-prescriptive step lists — instructions enumerating behaviors a current model handles from a brief instruction." A 9-step numbered git-PR procedure is a textbook example. Classification is correct. ✓

I8 tier handling. I8 is behavioral, which means proposals must pass the delete-and-watch verify loop before surfacing — never emitted as confident removals. The 4th expectation at line 74 enforces this directly. ✓

Expectation coverage. The four expectations exercise four orthogonal obligations:

#	What it tests
1	Classification — I8 over-prescriptive finding
2	Remediation — cull to intent + hard constraints, not verbatim preservation
3	Preservation nuance — genuine ordering / safety-gates / external-contract steps kept
4	Behavioral-tier gate — surfaced as verify-pass-vetted proposal, not unverified removal

The 4th expectation is a net addition over the other five cases (which have 3 each). This is the right call here — I8 is the only check in the catalog where both a specific content pattern and a behavioral-tier gating requirement are simultaneously at play in a single finding.

No collision with id 5 (verify-pass-demotion). id 5 exercises what happens when the verifier defends a prior-model instruction (demotion to info). id 6 exercises the initial classification and culling of a step-list finding. They are orthogonal scenarios. ✓

Schema compliance. id: 6 (unique integer), name matches ^[a-z0-9]+(-[a-z0-9]+)*$, files: [], all required fields present. ✓

---

plugin.json — version bump 0.9.0 → 0.9.1

Correct patch increment for a non-breaking eval-only addition. ✓

---

CHANGELOG.md — [0.9.1] entry

Entry follows Keep a Changelog ### Added convention, accurately records the lineage (absorbed from the superseded audit-model-fit C2 analog per #870), and matches the prior entry's style. ✓

---

No issues found

The Codex feedback was addressed. The case is schema-valid, correctly mapped to I8 (behavioral tier), the verify-pass gate is now an explicit expectation, and the housekeeping (version bump + CHANGELOG) is in order. The commit subjects follow Conventional Commits with the test(claude-config): scope.

Verdict: ready to merge.

[claude]

Claude finished @kyle-sexton's task in 1m 9s —— View job

---

Security Review — PR #892

• Gather context
• Fetch PR diff
• Read changed files
• Perform security analysis
• Post findings

---

Findings

No security issues found.

The two commits in this PR touch only static, non-executable data across three files.

File	Change	Security verdict
plugin.json	Version string bump 0.9.0 → 0.9.1	No risk — declarative metadata only
CHANGELOG.md	Prose documentation entry	No risk — documentation only
evals/evals.json	New eval case id 6 (HEAD also added a 4th expectation)	No risk — see analysis below

evals.json id 6 analysis:

• prompt — natural-language scenario text; no shell metacharacters used as commands, no template interpolation, no dynamic expansion. Consumed by an LLM eval harness as opaque input, not by a shell or query executor.
• expected_output — static descriptive string; no executable content.
• expectations — static string array with 4 entries; no code-execution paths.
• files — empty array []; no file-path references that could introduce path traversal.

Surfaces checked and found clean:

• Injection (command, SQL, path, template) — none; all changed values are inert strings.
• Secret / credential exposure — no tokens, API keys, or passwords.
• GitHub Actions patterns (pull_request_target, workflow_run, script injection via github context, permission widening) — no workflow files changed.
• Supply-chain risk (loosened or unpinned action/dependency pins) — no dependency manifests changed.
• Access-control or settings drift — no settings.json, MCP configs, or permission files touched.

This is a patch-level, data-only addition with no security surface.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7db35fcc18

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[claude]

Claude finished @kyle-sexton's task in 1m 24s —— View job

---

PR Review

• Gather context
• Read changed files (HEAD commit 76ae480)
• Assess Codex second feedback on step-list fixture
• Review for correctness, security, and guideline alignment
• Post findings

---

Summary

Three commits together build eval case id 6 (step-list-culled-not-preserved):

Commit	What it added
ccdc14e	Initial case: classify as I8, cull to intent + constraints, preserve nuance
7db35fc	4th expectation: behavioral-tier verify-pass gate (Codex P2 #1, accepted)
76ae480	Spell out all 9 steps in prompt; pin expectations to both sides of the cull (Codex P2 #2, accepted)

This review focuses on HEAD (76ae480), which is the only new commit since the last review.

---

evals/evals.json — id 6 after 76ae480

Codex second feedback status: resolved. The previous prompt showed only "run git status, then git add, then write the message, then..." — abbreviated to the point that a model could pass by generically asserting it would cull non-load-bearing steps without ever seeing the actual list. The new prompt enumerates all 9 steps explicitly:

1. run git status · 2. git diff · 3. git add · 4. write commit message · 5. run tests, stop on failure · 6. commit · 7. push · 8. title per Conventional Commits (CI enforces) · 9. gh pr create

Steps 5 and 8 are the load-bearing steps; 1–4, 6, 7, 9 are narration. A model can no longer pass with a generic answer — it must identify the right load-bearing steps by category.

Expectations after 76ae480 (5 total — all orthogonal):

#	What it tests
1	Classification — I8 over-prescriptive finding
2	Remediation — cull to intent + hard constraints, not verbatim
3	Preservation — test-gate (step 5) + Conventional Commits title (step 8) kept
4	Culling precision — narration steps (status/diff/add/commit/push) removed
5	Behavioral-tier gate — surfaces as verify-pass-vetted proposal, not confident removal

Expectation 3 replaces the previous generic "keeps steps encoding genuine ordering/safety-gate/external-contract" with two named concrete cases. This makes the regression surface much narrower and more testable. ✓

expected_output consistency. The new expected_output names both load-bearing steps by their substantive descriptions ("run-tests-and-stop-on-failure", "Conventional Commits PR-title requirement CI enforces") and names the narration steps to cull ("status/diff/add/message/commit/push"). This is fully consistent with expectations 3 and 4. ✓

Schema compliance. id: 6 unique, name pattern match, files: [], all required fields present. ✓

---

No issues found

All Codex feedback across both review passes has been addressed. The case is now concrete, machine-checkable, and covers five orthogonal obligations. The change is evals.json-only (5 ins, 4 del), no security surface, no schema violations, no housekeeping required (version and CHANGELOG were handled in ccdc14e).

Verdict: ready to merge.

[claude]

Claude finished @kyle-sexton's task in 1m 4s —— View job

---

Security Review — PR #892

• Gather context
• Fetch PR diff
• Read changed files
• Perform security analysis
• Post findings

---

Findings

No security issues found.

The three changed files are exclusively static, non-executable data:

File	Change	Security verdict
plugin.json	Version string 0.9.0 → 0.9.1	No risk — declarative metadata only
CHANGELOG.md	Prose documentation entry	No risk — documentation only
evals/evals.json	New eval case id 6 (+ 4th expectation in HEAD)	No risk — see analysis below

evals.json id 6 analysis:

The new case at lines 65–77 adds four fields:

• prompt — natural-language scenario text. Git command names appear in the prose (git status, git add, etc.) as descriptions of a workflow, not as shell expressions. The value is an inert string passed to an LLM eval harness as input; it is not executed by a shell or command interpreter. No injection risk.
• expected_output — static string describing expected LLM behavior. No executable content.
• files: [] — empty array. No file-path references; no path traversal surface.
• expectations — static string array. No code-execution paths.

Surfaces checked and found clean:

• Injection (command, SQL, path, template) — none; all changed values are inert strings consumed by an LLM, not a shell or query executor.
• Secret / credential exposure — no tokens, API keys, or passwords anywhere in the diff.
• GitHub Actions patterns (pull_request_target, workflow_run, script injection via github context, permission widening) — no workflow files changed.
• Supply-chain risk (loosened or unpinned action/dependency pins) — no dependency manifests changed.
• Access-control or settings drift — no settings.json, MCP configs, or permission files touched.

This is a patch-level, data-only addition with no security surface.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 76ae480b95

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Cover every culled step in this eval

Fresh evidence since the prior comment: the full 9-step list is now present, but this cull assertion still only requires removing status/diff/add/commit/push. A response can preserve step 4 (write a commit message) and step 9 (open the PR with gh pr create) while satisfying the objectively-verifiable expectations, even though line 73 names only steps 5 and 8 as load-bearing; that leaves the case unable to catch a partial step-list trim regression.

Useful? React with 👍 / 👎.