claude-ops: tune auto-mode classifier from observed team usage (step-3 guardrail, 1→2 residue)

[kyle-sexton]

Problem

The auto-mode permission classifier runs untuned defaults; nobody has fed observed usage
(which prompts recur, which denials were wrong) back into its configuration. Boris's step-3
guardrail names this verbatim: "Tune Auto mode classifier based on your team's usage" — and
WP5-Q8 explicitly routed this cell here rather than into the guardrail matrix.

Fix shape

Mine session telemetry / transcripts for recurring permission prompts and denied-then-manually-
approved patterns, then tune the classifier configuration (and the shared allowlist it
composes with) from that evidence. Recurring, not one-shot: candidate standing routine class
once the evidence pipeline exists. Relation: consumes the shared allowlist work and the
claude-ops observability lane; #455 documents one observed classifier-behavior mismatch.

Source

1→2 residue item from the AI-adoption-ladder gap map (step-3 guardrail cell, WP5-Q8
deferral); #239 directed "file via /work-items" — this files it.

---

Reference: Boris Cherny, "Steps of AI Adoption" (Google Doc)

[kyle-sexton]

Observed-usage mining + tuning delivered (2026-07-20)

Evidence (transcripts 2026-07-07→07-20 on the primary operator machine; 458 sessions with denials; 548 real permission prompts after excluding 614 PreToolUse guardrail-hook rejections that are not classifier prompts):

• Composition, not verbs, is the friction: 84% of shell prompts are compound commands (cd … && …, ;, |); of prompts whose operative verb was ALREADY allowlisted, 92% were compound — a leading cd defeats prefix-matching allow rules. More verb rules cannot fix this class.
• Strongest denied-then-approved signals: gh pr merge (26), git worktree remove (26), gh pr view (20), gh api repos (16), git fetch (14), gh pr create (13).
• One genuine rule-shape gap: the allowlist carried ZERO PowerShell() allow entries — all 47 PowerShell prompts fired by construction.

Tuning applied (the org-tunable surface): PowerShell read-parity rules (20) landed in the shared allowlist — the standards claude-permissions component (melodic-software/standards#210), consumed via the dotfiles composition seam (melodic-software/dotfiles#242). Merge verbs deliberately excluded from the floor (write-class, operator policy). Blanket compound-command allows rejected on safety evidence: 26 of 143 compound allowlisted-verb prompts carried a destructive tail (reset --hard, --no-verify, restore ., --force-with-lease).

Classifier-internal gaps (not org-tunable; recorded, deferred with trigger — vendor surface changes):

1. No compound-command decomposition — the classifier could auto-approve a compound iff every segment is individually safe; today the whole string prompts.
2. A prefix allow rule does not short-circuit the category safety layer for external writes (gh api repos/* / graphql / issue create allow rules still prompted in the final 48h of the window).
3. An autoMode.allow natural-language standing grant for org merges was not honored (39 "Merge Without Review" blocks despite the standing-approval text). source-control:babysit-prs: safety.md's 'never retry a harness permission denial' rule doesn't match auto-mode classifier's actual (retriable) behavior #455 documents an adjacent classifier-behavior mismatch.

Recurring-routine candidacy (per this issue's own framing): with standing telemetry emission now live (#352) plus the transcript store, the evidence pipeline exists; re-running this mining periodically is a candidate standing-routine class — held with the other dormant routine births (user-reserved, per the effort's trigger discipline).