Skip to content

feat: absorb boris-video digest — run-e2e config surface, gap report, recording tier, routines provenance#845

Merged
kyle-sexton merged 7 commits into
mainfrom
feat/boris-video-absorption
Jul 21, 2026
Merged

feat: absorb boris-video digest — run-e2e config surface, gap report, recording tier, routines provenance#845
kyle-sexton merged 7 commits into
mainfrom
feat/boris-video-absorption

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Absorbs the vetted ideas from the "4 Levels of Agentic Coding" video digest (Boris Cherny "Steps of AI Adoption" walkthrough) into the plugin estate, per the approved plan at docs/topics/boris-video-absorption/PLAN.md.

  • testing 0.3.0 (minor): /testing:run-e2e gains the consumer config surface .claude/testing/e2e.md (recording: video|gif|off default off; browser_mode: headed|headless default headless) with explicit read path, per-key merge + provenance reporting, and plumbing to the playwright executor; keys owned by new context/e2e-config.md (policy-vs-mechanics boundary; prompt > config precedence). Prerequisite hard-fail keeps STOP and adds a structured verification-environment gap report. Drive loop subagent-isolated; presence-gated /verify-first delegation (CC >=2.1.145). Evidence contract adds optional recording tier + session artifacts. Eval chore: initialize marketplace scaffold #1 updated.
  • verification 0.3.2 (patch): /verification:confirm delegation wording acknowledges the enriched run-e2e; stages/verdict machinery untouched.
  • autonomy 0.8.0 (minor): reference/routines.md gains the instruction-provenance clause — routine instruction content lives in a version-controlled artifact, stored prompt is a thin pointer; surface mappings are illustrative deployment-owned bindings.
  • Implementers-table row added in docs/conventions/consumer-config-layering/README.md.

Coordination: zero file overlap with the ignition/enforcement lanes (#778 comment posted; ignition Phases 2–3 merged via #810). Defaults preserve current consumer behavior.

Related

No linked issue: this PR closes no GitHub issue. Related, not closed:

Verification

  • markdownlint: 0 errors on all touched files; typos gate exit 0
  • check-skill.sh: run-e2e PASS, confirm PASS (pre-existing advisory warnings only)
  • run-e2e evals/evals.json: valid against bundled schema (ajv, draft2020)
  • Sanity greps per PLAN phase checks: all pass
  • Stage-structure invariant on confirm SKILL.md: 16 before/after
PLAN.md

PLAN — boris-video-absorption

Brief

Absorb the vetted ideas from the Ray Amjad "4 Levels of Agentic Coding"
video digest (research base:
~/.work/youtube-watch/the-4-levels-of-agentic-coding-how-to-sh-XLA-sTSJ-Wc/)
into the plugin estate. Interview-locked contracts (this session):

  • Extend existing seams, no new skill: testing:run-e2e (driver) +
    verification:confirm (judge) keep their division of labor.
  • run-e2e gains: subagent-isolated surface runs; native-/verify-first
    delegation (presence-gated, CC ≥2.1.145); structured missing-environment
    gap report on prerequisite failure (hard-fail stays, plus actionable
    report); video/session-artifact evidence tier.
  • Recording format (video | gif | off) and browser mode (headed |
    headless) are layered config knobs per
    docs/conventions/consumer-config-layering/ — NOT hardcodes.
  • plugins/autonomy/reference/routines.md gains a thin-pointer clause:
    routine/scheduled-task prompts on any surface point to
    version-controlled files; pasted prose non-compliant.
  • ~/.claude/scheduled-tasks/ (exists, live task) proposed for dotfiles
    tracking via the dotfiles repo's add-dotfile flow.
  • Deferred-with-trigger (recorded in digest slice, NO tracker issues):
    verification self-improvement loop (trigger: accumulated recorded
    runs); bounded build→review→fix loop (trigger: post-Phase-II
    enforcement rail); Claude Tag enablement (trigger: post-8/3 cutover +
    Enterprise admin check).
  • Out of scope: presenter-custom harness (Percy), chat-surface wiring,
    Chronicle adoption, lifecycle reordering.
  • Boris doc capture re-verified NO-DRIFT 2026-07-21; all adopted items
    trace to the doc's step 1→2/2→3 recipes or native CC features.

Success: PRs merged with skill-quality + markdownlint green; every
adopted behavior traceable to research findings; zero collision with the
primary session's lanes (ignition Phases 2–3 merged via #810; no pending
branch touches target files — verified).

Standards grounding

  • docs/conventions/consumer-config-layering/README.md — three layers
    (user-global / team / local overlay), additive-preferred merge; new
    config surface must declare keys in the owning plugin's docs and point
    at the convention for layering.
  • docs/CATALOG-TAXONOMY.md + docs/PLUGIN-PHILOSOPHY.md — extend-over-
    add, seam cooperation presence-gated, fresh-eyes delegation.
  • Repo commit/PR conventions (conventional commits, codex thread
    resolution before merge, --body-file - for PR bodies).

Plan

Phase 1: Lane setup [DONE]

  • Create worktree off origin/main, branch feat/boris-video-absorption
    (sibling-worktree convention).
  • Materialize docs/topics/boris-video-absorption/PLAN.md (this file) +
    design/design-resolution.md (Tier B early-exit: no new types; one new
    config surface whose shape is resolved here — see Decisions).
  • Post coordination comment on autonomy: ignite the 2-to-3 loop — first standing routine (hourly C2 drain) + C2-promotion evidence drain on the proving ground #778: secondary session lane, files
    touched, no overlap with ignition/enforcement lanes.
  • Sanity Check: git worktree list shows the new worktree on the new
    branch; gh issue view 778 shows the comment; PLAN.md exists in the
    worktree at docs/topics/boris-video-absorption/PLAN.md.

Phase 2: run-e2e enrichment [DONE]

Files: plugins/testing/skills/run-e2e/SKILL.md,
plugins/testing/skills/run-e2e/context/e2e.md,
plugins/testing/README.md, new
plugins/testing/skills/run-e2e/context/e2e-config.md (key owner doc),
plugins/testing/skills/run-e2e/evals/evals.json (eval #1 asserts the
prerequisite-failure behavior this phase changes — update
unconditionally), plugins/testing/CHANGELOG.md (Keep a Changelog entry;
minor bump — new behaviors),
docs/conventions/consumer-config-layering/README.md (Implementers-table
row for the new surface — the convention tracks conformance, not
assumes it).

  • Declare config surface .claude/testing/e2e.md (folder-form name per
    convention: surface identity = whole path relative to .claude/).
    Keys: recording: video | gif | off (default off — evidence contract
    screenshots stay the floor), browser_mode: headed | headless (default
    headless, operator overrides for visibility). Per-key override
    semantics. Layering resolution cites the convention doc — no
    restatement.
  • Read path (reviewer gap feat: markdown-formatter plugin (pilot migration) #3 — mandatory): SKILL.md gains explicit
    resolution instructions — anchor at repo root, read all three layers
    (user-global → team → local overlay), merge per-key, report which layer
    supplied each effective value. Plumbing: resolved browser_mode
    and recording are passed to the executor (/playwright:playwright
    invocation args / gif_creator choice) — keys without plumbing are the
    documented ai-briefing deviation; not repeating it.
  • Key-ownership framing (reviewer risk perf(markdown-formatter): unwired producer builds telemetry payload before the opt-in guard (~90ms/edit on Windows) #6): e2e-config.md states the
    boundary — run-e2e owns capture POLICY (what evidence, which format,
    visibility preference); playwright owns MECHANICS (how the browser
    runs). Keys are policy inputs run-e2e resolves and passes through.
  • Precedence (operator-locked): config keys are DEFAULTS only; an
    explicit session prompt always wins ("run this headed" overrides
    browser_mode: headless). e2e-config.md states this: prompt >
    local overlay > team > user-global > bundled default.
  • SKILL.md: subagent-isolated run guidance (delegate the drive loop to a
    subagent; orchestrator consumes evidence paths only); /verify-first
    delegation note (presence-gated on bundled /verify, CC ≥2.1.145, with
    existing orchestrator fallback unchanged).
  • Prerequisite failure path: keep hard-fail STOP; add a structured
    verification-environment gap report (what's missing, what the operator
    must provide — keys/CLIs/environments; the "what do you need to verify
    this e2e" elicitation) written to the run's evidence output.
  • context/e2e.md evidence contract: add optional recording tier (video
    via playwright CLI preferred for long flows; GIF via gif_creator for
    short demos — driven by the recording key) + session artifacts row
    (recording path, session ID, transcript pointer) in the evidence table.
  • Sanity Check:
    rg -c "e2e-config|\.claude/testing/e2e\.md" plugins/testing/skills/run-e2e/SKILL.md
    ≥ 1; rg -c "gap report" plugins/testing/skills/run-e2e/SKILL.md ≥ 1;
    rg -c "recording" plugins/testing/skills/run-e2e/context/e2e.md ≥ 2;
    read-path present:
    rg -c "layer" plugins/testing/skills/run-e2e/SKILL.md ≥ 2 AND
    rg -c "browser_mode" plugins/testing/skills/run-e2e/SKILL.md ≥ 2
    (resolution + plumbing sites); Implementers-table row:
    rg -c "testing/e2e" docs/conventions/consumer-config-layering/README.md
    ≥ 1; CHANGELOG entry present; eval chore: initialize marketplace scaffold #1 updated
    (rg -c "gap" plugins/testing/skills/run-e2e/evals/evals.json ≥ 1);
    markdownlint exit 0 on changed files; /skill-quality:check run-e2e
    PASS.

Phase 3: confirm delegation touch-up [DONE]

Files: plugins/verification/skills/confirm/SKILL.md,
plugins/verification/README.md (only if wording drifts),
plugins/verification/CHANGELOG.md (patch bump — wording touch-up).

  • Delegation section: acknowledge enriched run-e2e (subagent isolation +
    recording/session evidence tier + gap report); /verify stays named as
    the supplementary live-run path with unchanged presence gate. No
    structural change to stages or verdict machinery.
  • Sanity Check:
    rg -c "gap report|recording" plugins/verification/skills/confirm/SKILL.md
    ≥ 1; stage structure unchanged
    (rg -c "Stage" plugins/verification/skills/confirm/SKILL.md equal
    before/after); markdownlint exit 0.

Phase 4: routines.md thin-pointer clause [DONE]

Files: plugins/autonomy/reference/routines.md,
plugins/autonomy/CHANGELOG.md (minor bump — new normative clause).

  • New subsection, phrased to honor the doc's hosting-agnosticism
    (reviewer risk docs(playbook): document the local plugin development loop #7): the NORMATIVE clause is surface-agnostic — "a
    routine's instruction content lives in a version-controlled, reviewable
    artifact; the stored prompt is a thin pointer to it; pasted-prose
    prompts are non-compliant (no history, invisible drift)". The
    surface-specific mappings (cloud routines → committed
    .claude/skills/, documented transfer path; Desktop scheduled tasks →
    ~/.claude/scheduled-tasks/<task>/SKILL.md under dotfiles) appear only
    as ILLUSTRATIVE binding examples, marked as deployment-owned bindings
    per the doc's own Hosting stance. Cite research
    (routines-versioning-deep-dive.md: native prompt versioning
    documented-absent). If the clause reads better in
    wiring-vs-advisor.md at build time, the builder flags it back to the
    main thread rather than deciding — placement objection is the known
    review risk.
  • Sanity Check:
    rg -c "thin pointer|version-controlled" plugins/autonomy/reference/routines.md
    ≥ 2; agnosticism preserved:
    rg -c "deployment-owned" plugins/autonomy/reference/routines.md
    count increases by ≥1; markdownlint exit 0; CHANGELOG entry present.

Phase 5: digest-slice deferred items + through-line [DONE]

Files (home slice, not repo):
~/.work/youtube-watch/the-4-levels-of-agentic-coding-how-to-sh-XLA-sTSJ-Wc/recommendations/menu.md,
README.md (same slice).

  • Update menu items to final dispositions (adopted → PR refs; deferred →
    named triggers; no-go unchanged). Add through-line note: Boris doc →
    interview locks → PLAN → PRs.
  • Sanity Check: rg -c "deferred" menu.md ≥ 3; each adopted item
    carries a PR/branch reference.

Phase 6: dotfiles tracking proposal [TODO — user gate]

  • Run the dotfiles repo's add-dotfile flow for
    ~/.claude/scheduled-tasks/ (dir exists with live
    autonomy-demo-hourly-drain/). Flow owns chezmoi mechanics; never
    chezmoi apply from agent context.
  • Sanity Check: dotfiles repo PR/branch exists containing the
    scheduled-tasks source state, or an explicit operator decline recorded
    in PLAN.
  • User gate: surfaces before executing (cross-repo write).

Phase 7: verify + ship [TODO]

  • Empirical verification in worktree: markdownlint sweep, typos gate,
    /skill-quality:check per touched skill, eval validation.
  • Version bumps via Edit on the version line (biome retab gotcha).
  • Conventional commits; PR(s) with PLAN.md in <details>; codex threads
    resolved before merge; close-out per /planning:plan close-out
    (slice prune + pointer).
  • Sanity Check: CI green on PR; gh pr view shows mergeable; all
    phase tags [DONE].

Phase 8: trial run (post-merge validation) [TODO]

  • After plugins update, run the enriched flow end-to-end on a real UI
    change in a consumer repo: config surface resolved + reported,
    recording produced per key, gap report exercised (remove a prereq
    deliberately), evidence table carries session artifacts.
  • Sanity Check: recording artifact exists on disk; gap report emitted
    on induced prereq failure; operator reviews the recording.

Blast radius

LOW-MEDIUM. Markdown skill/contract text + one new additive config
surface; zero runtime code; three plugins touched; consumers unaffected
by default (recording defaults off, browser_mode default preserves
current behavior). Cross-repo edge: dotfiles proposal (user-gated).

Stress-test summary

Fresh-context plan-reviewer ran against live repo. 3 CONFIRMED-GAPs
folded: (1) phantom config surface — read-path resolution + executor
plumbing now mandated in P2 with sanity greps; (2) Implementers-table
row added to P2 files; (3) CHANGELOG entries added to P2/P3/P4 with
SemVer magnitudes (testing minor, verification patch, autonomy minor).
Risks addressed: key-ownership boundary framed policy-vs-mechanics in
e2e-config.md; routines.md clause re-phrased surface-agnostic with
bindings as illustrative examples; eval #1 update made unconditional.
Collision re-verified clean (no open PR overlap). Formal
/devils-advocate skipped: blast radius LOW-MEDIUM, no
infra/CI/dependency triggers.

Execution shape

Wave A (parallel, file-disjoint, one worktree, agents never touch git):

Phase Surface Basis
2 run-e2e Opus builder subagent mechanical doc edits, largest item
3 confirm Opus builder subagent small disjoint edit
4 routines Opus builder subagent small disjoint edit

Wave B (sequential, main thread): 5 (home slice), 6 (user-gated
cross-repo), 7 (verify + ship — main thread only, empirical checks +
git).

Phase 1 precedes all. Scope fences: each builder gets an ALLOWED list =
its phase's files; FORBIDDEN = PLAN.md, other phases' files, any git
command. Cost: 3 builders vs sequential saves modest wall-clock; fallback
= run 2→3→4 sequentially in main thread on any fence violation.

Open questions

None blocking — all interview-locked.

Handoff to implementation

User-approval gates

  • Phase 6 dotfiles add-dotfile run (cross-repo write).
  • Any mid-flight scope change to config-surface shape.

Execution shape ([EXEC-SHAPE] tagged)

  • [EXEC-SHAPE] PLAN drafted in .work/ first; materialized to
    docs/topics/ inside the worktree at Phase 1 (avoids writing to the
    canonical checkout the primary session owns).
  • [EXEC-SHAPE] Wave A 3-builder parallel shape with scope fences as
    above; sequential fallback documented.
  • [EXEC-SHAPE] Single PR for phases 2–4 (one review surface, shared
    rationale) unless review load argues for a split at ship time.

Mechanical work

  • Commit boundaries: Phase 1 (PLAN + design-resolution), one commit per
    phase 2–4, Phase 5 lives outside the repo (no commit), Phase 7 ship.
  • Verification checkpoints: markdownlint + typos + skill-quality per
    phase before its commit; empirical greps per Sanity Checks.
  • Sequential fallback: any fence violation → abandon parallel, re-run
    remaining phases in main thread.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QdCmaHa15bwUTh9sy7Rfdz

kyle-sexton and others added 5 commits July 21, 2026 05:29
…tion

Approved plan for absorbing vetted ideas from the "4 Levels of Agentic
Coding" video digest: run-e2e enrichment (config surface, gap report,
recording tier), confirm delegation touch-up, routines.md thin-pointer
clause. Coordination comment posted on #778.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdCmaHa15bwUTh9sy7Rfdz
The live-app delegation section now names run-e2e's subagent-isolated
drive loop, optional recording/session-artifact evidence tier, and
structured gap report on prerequisite failure. /verify supplementary
path and stage/verdict machinery unchanged. Patch entry [0.3.2].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdCmaHa15bwUTh9sy7Rfdz
Routine instruction content lives in a version-controlled, reviewable
artifact; the stored prompt is a thin pointer. Pasted-prose prompts are
non-compliant (no history, invisible drift). Cloud-routine and Desktop
scheduled-task mappings appear as illustrative deployment-owned
bindings, consistent with the Hosting stance. Minor entry [0.8.0].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdCmaHa15bwUTh9sy7Rfdz
run-e2e gains the consumer config surface .claude/testing/e2e.md
(recording: video|gif|off default off; browser_mode: headed|headless
default headless) with an explicit read path, per-key merge, provenance
reporting, and plumbing of resolved values to the playwright executor.
Keys are owned by the new context/e2e-config.md (policy-vs-mechanics
boundary, prompt-over-config precedence ladder). The prerequisite
hard-fail keeps its STOP and now also writes a structured
verification-environment gap report. The drive loop is
subagent-isolated; verification delegates to bundled /verify when
present (CC >=2.1.145). Evidence contract adds an optional recording
tier and session-artifacts rows. Eval #1 asserts the new
prerequisite-failure shape. Implementers table gains the surface row.
Minor entry [0.3.0].

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdCmaHa15bwUTh9sy7Rfdz
Aligns manifests with the CHANGELOG entries from the
boris-video-absorption phases; PLAN phase tags updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdCmaHa15bwUTh9sy7Rfdz
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@claude

claude Bot commented Jul 21, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 2m 54s —— View job


PR Review — feat/boris-video-absorption

Tasks

  • Load review tooling
  • Read changed files
  • Review testing plugin changes (run-e2e SKILL.md, e2e-config.md, e2e.md, evals)
  • Review verification plugin changes (confirm SKILL.md)
  • Review autonomy plugin changes (routines.md)
  • Review docs/conventions changes
  • Review plugin.json version bumps
  • Post findings

Overview

Solid, well-structured absorption. Defaults are non-breaking, semver magnitudes are correct, and the three CHANGELOG entries accurately describe the changes. The PLAN.md's stated sanity checks all pass against the actual files. Three findings worth addressing before merge:


Findings

1 — Raw githubusercontent.com URL is repo-coupled (Moderate)

SKILL.md line 41 and e2e-config.md line 5 both link:

[consumer-config layering contract](https://github.com/ghraw/melodic-software/claude-code-plugins/main/docs/conventions/consumer-config-layering/README.md)

This hardcodes a raw GitHub URL to this specific repo. If the repository is renamed or forked, these links break silently in installed plugins — which run from the isolated cache, not from a live clone. Every other cross-context link in the skill suite uses a relative path within the plugin ([context/e2e.md](context/e2e.md)). The convention doc isn't inside the plugin, so a relative path can't reach it, but the prose can still name the convention by path without hyperlinking to a remote URL. Referencing the convention as "docs/conventions/consumer-config-layering/README.md in the plugin marketplace" gives reviewers and readers the same signal without coupling to a specific repo hostname.

Fix this →


2 — e2e.md "Capture evidence" section not reconciled with new recording tier (Low)

e2e.md lines 128–131 (pre-existing, not touched by this PR) still reads:

For multi-step flows, use Claude in Chrome GIF recording:
mcp__claude-in-chrome__gif_creator → record the interaction sequence

The new Recording tier section (lines 54–60, added by this PR) describes gif as the config-driven path realized through gif_creator. The two descriptions now coexist without reconciliation: the pre-existing text reads as a standalone manual instruction while the new tier describes it as config-driven. A model executing Step 3 could follow both paths or the wrong one. A brief cross-reference ("see the Recording tier above") or a qualifier ("when recording: gif is set") on the old paragraph would resolve the ambiguity.

Fix this →


3 — Handoff /verify presence note lacks the provenance date that confirm carries (Low)

SKILL.md Handoff, line 57:

when the bundled /verify command is present (Claude Code ≥2.1.145)

The corresponding line in confirm SKILL.md line 108 carries the verification date and source:

requires Claude Code ≥2.1.145 — verified 2026-07-18 against [bundled skills](https://code.claude.com/docs/en/skills#bundled-skills)

The run-e2e reference is the same capability claim but without the provenance. Minor inconsistency — future readers of SKILL.md can't tell when the version floor was verified or from what source.


What's well done

  • Defaults preserve current behavior exactly. recording: off and browser_mode: headless mean consumers with no config surface see zero behavioral change.
  • Policy-mechanics separation in e2e-config.md is correctly framed. run-e2e owns capture policy, /playwright:playwright owns the flag mechanics. This is the right boundary for layered plugins.
  • Thin-pointer clause is surface-agnostic. The routines.md addition avoids locking to any scheduling surface, marks both bindings (cloud + Desktop) as illustrative and deployment-owned — exactly what the PLAN required and reviewer risk docs(playbook): document the local plugin development loop #7 called out.
  • Semver magnitudes are correct. testing 0.2.5→0.3.0 (new behaviors), verification 0.3.1→0.3.2 (doc-only wording), autonomy 0.7.4→0.8.0 (new normative clause). All aligned with Keep a Changelog semantics.
  • CHANGELOG entries are accurate and complete. Each entry correctly describes what changed without overstating or understating scope.
  • Implementers-table row matches actual implementation. All three layers, per-key override, key ownership correctly cited — the row would pass a conformance audit against the files it describes.
  • Eval chore: initialize marketplace scaffold #1 update is correct. The updated expected_output accurately captures the new prerequisite-fail path: STOP + gap report, then proceed with config resolution. The structure matches the SKILL.md's Step 1 additions.

@claude

claude Bot commented Jul 21, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 2m 57s —— View job


Security Review — PR #845

  • Load tools
  • Fetch PR diff
  • Read changed files
  • Analyze for security vulnerabilities
  • Post findings

Scope

This PR is 491 additions / 5 deletions across 16 files — all markdown prose and JSON manifests. No GitHub Actions workflows, no executable code, no secret handling, no dependency changes.


Finding 1 — Unpinned live-branch URL references inside AI skill instructions

Severity: SUGGESTION
Confidence: CONFIRMED (references exist; exploitability depends on whether the AI fetches the URL during a run)

Two newly-added files instruct the AI to consult an external URL for layering-contract mechanics:

Both reference:

https://github.com/ghraw/melodic-software/claude-code-plugins/main/docs/conventions/consumer-config-layering/README.md

The URL resolves to the live main branch, not a pinned commit SHA. This creates a version-decoupling risk specific to AI skill systems: when the AI agent executing a testing@0.3.0 plugin run encounters a "see layering contract" instruction and issues a WebFetch, it retrieves whatever content main holds at that moment — not the content that was current when version 0.3.0 shipped. A change to the layering convention on main (including a malicious one if the branch were compromised) would immediately alter runtime behavior across all deployed plugin versions without a version bump or consumer notification. The file is in the same repo (lower exfiltration risk) but the pinning guarantee the plugin's semver communicates is silently violated.

This is qualitatively different from a markdown hyperlink for human readers: inside an AI skill document, a "see [URL]" with a full github.com/ghraw address is an actionable fetch instruction.

Suggested fix: replace main with the commit SHA at merge time (or a released tag ref), e.g.:

https://github.com/ghraw/melodic-software/claude-code-plugins/<SHA>/docs/conventions/consumer-config-layering/README.md

Alternatively, reference the file via ${CLAUDE_PLUGIN_ROOT} or a relative path so the plugin's own installed copy is consulted rather than a live remote URL.


Finding 2 — Consumer config surface as AI-readable prompt-injection vector

Severity: SUGGESTION
Confidence: PLAUSIBLE (inherent to the design; not unique to this PR)

The new surface .claude/testing/e2e.md is read by the AI as text and merged into its instruction set (Step 2 of SKILL.md). A consumer repository hosting a maliciously crafted config file — containing embedded prompt-injection text alongside the valid YAML keys — could influence the AI's behavior beyond the intended recording / browser_mode values. Example: a file that sets recording: off but then appends adversarial prose that overrides later skill instructions.

This risk is present for every consumer-config surface in this marketplace (it is inherent to AI-readable config), so it is not newly introduced by this PR. The finding is noted here because this PR adds a new surface, expanding the attack area. The closed scalar values and short key-set limit the viable injection window somewhat, but the merge step reads the full file text rather than parsing a strict schema.

No code change is required to "fix" this — the architecture is AI-skill design; the mitigations are operational (validate consumer config files in CI, keep surfaces narrow and schema-validated). Noting it so the team is aware the new surface inherits this property.


No other security issues found

  • No GitHub Actions workflows modified — no pull_request_target, script-injection, or permission-widening risks.
  • No secrets, credentials, or tokens handled or referenced.
  • No new dependency pins or supply-chain changes (version bumps are only in plugin.json version fields).
  • Pre-computed context shell blocks in skill files (! prefix — git status, git diff, git log, git branch) are pre-existing in both run-e2e/SKILL.md and confirm/SKILL.md and are not modified by this PR.

The plugin contract gate bans vendor names in reference/ contracts;
the illustrative bindings now name surface classes (cloud scheduling
surface, desktop scheduling surface) instead of products and
vendor-coupled paths. CHANGELOG wording aligned.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdCmaHa15bwUTh9sy7Rfdz
@claude

claude Bot commented Jul 21, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 2m 48s —— View job


Security Review — PR #845

Tasks

  • Load tools
  • Fetch PR diff (via git log + direct file reads)
  • Read changed skill files for injection surfaces
  • Check for dangerous shell commands / pre-computed context blocks
  • Check for hardcoded URLs / supply-chain risks
  • Check for secret / credential handling issues
  • Check for GitHub Actions patterns
  • Post findings

Scope

16 files — all markdown skill documents and plugin.json semver bumps. No GitHub Actions workflows, no executable code, no secrets, no dependency pins.

This synchronize trigger adds one commit beyond the prior security review (fix(autonomy): phrase provenance bindings as surface classes — a prose-only refinement of the instruction-provenance clause in routines.md). That commit is covered below; no new surface is introduced by it.


Finding 1 — Unpinned live-branch URL inside AI-executable skill instructions

Severity: SUGGESTION
Confidence: CONFIRMED

Two files added by this PR embed a github.com/ghraw link pointing at the live main branch:

https://github.com/ghraw/melodic-software/claude-code-plugins/main/docs/conventions/consumer-config-layering/README.md

The main ref is live and unpinned. Inside an AI skill document this is not merely a human-readable link — an agent running a testing@0.3.0 skill that issues a WebFetch on this URL receives whatever content main holds at execution time, not what was current at 0.3.0 release. The version-decoupling is silent: a future change (or compromise) of the layering contract on main would take effect in all deployed plugin versions without a semver bump or consumer notification. The fetch surface is also a prompt-injection entry point: injected content arriving via this URL enters the agent's instruction context during execution.

Risk is moderated because the URL resolves within the same repository (no third-party trust involved), but the semver isolation the plugin's version number implies is violated.

Suggested fix: replace main with the merge-commit SHA, e.g.:

https://github.com/ghraw/melodic-software/claude-code-plugins/<SHA>/docs/conventions/consumer-config-layering/README.md

Or drop the hyperlink entirely and reference the path as prose (docs/conventions/consumer-config-layering/README.md in the plugin marketplace repo) — the prior code-review comment on Finding 1 covers this alternative.

Fix this →


Finding 2 — New consumer config surface is an AI-readable prompt-injection vector

Severity: SUGGESTION
Confidence: PLAUSIBLE

The new surface .claude/testing/e2e.md — added by this PR — is read as markdown text and merged into the agent's instruction context (SKILL.md Step 2). A malicious or misconfigured consumer config file could embed adversarial prose alongside the valid recording / browser_mode key-value pairs, influencing agent behavior beyond the intended keys. The skill reads the full file text rather than parsing a strict machine-readable schema.

This risk is inherent to any AI-readable config surface and is not unique to this PR; the closed scalar values (video|gif|off, headed|headless) and short key-set limit the viable injection window. The finding is noted because this PR introduces a new surface, expanding the attack area. Exploitation requires write access to a consumer repo's .claude/ directory — an insider or CI-pipeline threat model, not a remote one.

No code change is required. Existing mitigations (narrow key-set, defaults off) are correct. Teams operating in high-trust-boundary environments may wish to validate consumer config files in CI before use.


Delta commit — fix(autonomy): phrase provenance bindings as surface classes

The commit added since the prior security review is a prose-only refinement of the instruction-provenance clause in plugins/autonomy/reference/routines.md. It rephrases illustrative deployment bindings as surface-class examples rather than named-tool specifics. No new execution surface, URL, shell command, or config key is introduced. No new security findings.


No other security issues found

  • Pre-computed context blocks (! backtick shell commands) in both run-e2e/SKILL.md and confirm/SKILL.md are pre-existing, unmodified by this PR. Both are read-only git introspection commands with no user-controlled input — no injection risk.
  • GitHub Actions — no .github/workflows/ files modified.
  • Secrets / credentials — none referenced or handled.
  • Supply chain / dependency pins — version bumps are plugin.json version fields only (semver strings, not package pins or action refs).
  • CLAUDE_PLUGIN_ROOT / CLAUDE_PLUGIN_DATA isolation — new context files are bundled within the plugin at ${CLAUDE_PLUGIN_ROOT}/skills/run-e2e/context/; no ../ reach-outs.

@claude

claude Bot commented Jul 21, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 41s —— View job


PR Review — feat/boris-video-absorption

Tasks

  • Load review tooling
  • Read changed files (current HEAD: 7612802)
  • Check status of prior findings
  • Review autonomy changes from latest commit (fix(autonomy): phrase provenance bindings as surface classes)
  • Post findings

Latest commit assessment

The most recent commit (7612802) is a clean fix: it replaces hard-coded product names ("Cloud routines", "Claude Desktop scheduled tasks", ~/.claude/scheduled-tasks/<task>/SKILL.md) with surface-class descriptions ("A cloud scheduling surface whose runs clone selected repositories", "A desktop scheduling surface with per-task instruction files"). The corresponding CHANGELOG entry was aligned in the same commit. This satisfies the PLAN's hosting-agnosticism requirement (reviewer risk #7) and the plugin contract's ban on vendor names in reference/ contracts.


Findings — three prior items remain unaddressed

The three findings from the earlier review are all still present in the current HEAD. Noting them here for the record since they were not addressed in the intervening push.

1 — Raw live-branch URL couples installed plugin to a specific GitHub hostname (Moderate)

Both files still reference:

https://github.com/ghraw/melodic-software/claude-code-plugins/main/docs/conventions/consumer-config-layering/README.md

This is a live main URL — unpinned. Two concerns: (1) if the repo is renamed or forked, the link breaks silently for all consumers running the installed plugin; (2) inside an AI skill document a github.com/ghraw link is an actionable fetch instruction — the agent may retrieve whatever main holds at run time, decoupling runtime behavior from the pinned plugin version. The convention doc is not inside the plugin, so a relative path can't reach it — but the prose can name the path without hyperlinking remotely. Replacing the link with docs/conventions/consumer-config-layering/README.md (in the plugin marketplace) gives reviewers and the AI the same navigational signal without coupling to a hostname.

Fix this →


2 — Old GIF instruction in e2e.md not reconciled with new Recording tier (Low)

e2e.md lines 128–131 (pre-existing, untouched by this PR) still reads:

For multi-step flows, use Claude in Chrome GIF recording:
mcp__claude-in-chrome__gif_creator → record the interaction sequence

The new Recording tier section (lines 52–60, added by this PR) now owns gif as the config-driven path via gif_creator. The two descriptions coexist without reconciliation: the Step 5 text reads as a standalone always-do instruction while the new tier describes it as a config-driven option. A model executing Step 5 could treat both as independent paths and run gif_creator regardless of the recording key. A cross-reference or qualifier ("when recording: gif is resolved from config, capture via gif_creator; see the Recording tier above") would close the ambiguity.

Fix this →


3 — Handoff /verify version note lacks provenance date (Low)

SKILL.md line 57:

when the bundled /verify command is present (Claude Code ≥2.1.145)

The same version floor appears in confirm/SKILL.md line 108 with a verification date and source:

requires Claude Code ≥2.1.145 — verified 2026-07-18 against [bundled skills](https://code.claude.com/docs/en/skills#bundled-skills)

Adding the same provenance to the run-e2e SKILL.md brings the two references into parity. Minor inconsistency but worth closing before merge.


What's working well

  • Latest commit is correct. Surface-class phrasing in routines.md is cleanly vendor-agnostic; CHANGELOG aligned. The fix earns the fix(autonomy) scope.
  • Defaults are non-breaking. recording: off and browser_mode: headless mean consumers with no config surface see zero behavioral change.
  • Thin-pointer clause is surface-agnostic. routines.md addition avoids locking to any scheduling surface, marks both bindings as illustrative and deployment-owned — exactly what PLAN reviewer risk docs(playbook): document the local plugin development loop #7 required.
  • Semver magnitudes are correct. testing 0.3.0 (new behaviors), verification 0.3.2 (doc-only), autonomy 0.8.0 (new normative clause).
  • Policy-mechanics boundary in e2e-config.md is correctly framed. run-e2e owns capture policy; /playwright:playwright owns flag mechanics. Right call.

Three prior findings remain open (one moderate, two low). Finding #1 is worth resolving before merge; findings #2 and #3 are low-risk but would improve internal consistency.

@claude

claude Bot commented Jul 21, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 35s —— View job


PR Review — feat/boris-video-absorption

Tasks

  • Load review tooling
  • Read latest commit changes (312f95d — docs(topics): mark phases 6-7 done)
  • Check status of three open findings from prior reviews
  • Review changed files at HEAD
  • Post findings

Latest commit assessment

312f95d (docs(topics): mark phases 6-7 done — dotfiles PR #270, ship complete) touches only docs/topics/boris-video-absorption/PLAN.md — 2 insertions, 2 deletions updating phase 6 and 7 tags to [DONE]. No plugin files, no skill documents, no config surfaces touched. No new findings.


Open findings — still unaddressed at HEAD

All three findings raised in prior reviews remain open in the current HEAD. Summarized below with current file + line permalinks.

1 — Raw live-branch URL inside AI-executable skill instructions (Moderate)

Two files still link directly to main:

SKILL.md line 41

see the [consumer-config layering contract](https://github.com/ghraw/melodic-software/claude-code-plugins/main/docs/conventions/consumer-config-layering/README.md)

e2e-config.md line 5

see the [consumer-config layering contract](https://github.com/ghraw/melodic-software/claude-code-plugins/main/docs/conventions/consumer-config-layering/README.md)

Inside an AI skill document a github.com/ghraw URL pointing at main is an actionable fetch instruction, not just a human hyperlink. An agent running testing@0.3.0 that fetches this URL will receive whatever main holds at execution time — silently decoupling plugin version from runtime behavior. Every other cross-context link in this skill suite uses a relative path (context/e2e.md, context/e2e-config.md). Since the convention doc sits outside the plugin, a relative path can't reach it, but the prose can name the path without a live URL: replacing the hyperlink with docs/conventions/consumer-config-layering/README.md (in the plugin marketplace) gives readers and agents the same navigational signal without coupling to a hostname or live branch.

Fix this →


2 — Pre-existing GIF instruction not reconciled with new Recording tier (Low)

e2e.md lines 128–131 (pre-existing, untouched by this PR):

For multi-step flows, use Claude in Chrome GIF recording:
mcp__claude-in-chrome__gif_creator → record the interaction sequence

This PR added a Recording tier section (lines 52–60) that makes gif_creator the mechanism selected by recording: gif. The two now coexist without reconciliation: the Step 5 paragraph reads as an always-do unconditional instruction while the Recording tier describes it as config-driven. A qualifier — e.g. when recording: gif is set (see Recording tier above), — on the Step 5 paragraph closes the ambiguity cleanly without restructuring either section.

Fix this →


3 — Handoff /verify presence note lacks provenance date (Low)

SKILL.md line 57:

when the bundled /verify command is present (Claude Code ≥2.1.145)

The identical version floor in confirm/SKILL.md line 108 carries full provenance:

requires Claude Code ≥2.1.145 — verified 2026-07-18 against [bundled skills](https://code.claude.com/docs/en/skills#bundled-skills)

Adding — verified 2026-07-18 against [bundled skills](https://code.claude.com/docs/en/skills#bundled-skills) to the run-e2e reference brings the two claims into parity and lets future readers trace when and where the floor was checked.


What's well done

  • Latest commit is clean. Phase-status update to PLAN.md only; no new surfaces or concerns introduced.
  • Defaults are non-breaking. recording: off and browser_mode: headless preserve current consumer behavior exactly.
  • Policy-mechanics boundary is correctly framed. run-e2e owns capture policy; /playwright:playwright owns flag mechanics. The boundary in e2e-config.md is right.
  • Thin-pointer clause is surface-agnostic. The routines.md addition (including the fix(autonomy) surface-class refinement) avoids locking to any scheduling surface or vendor name — exactly what reviewer risk docs(playbook): document the local plugin development loop #7 required.
  • Semver magnitudes are correct. testing 0.3.0, verification 0.3.2, autonomy 0.8.0 — all aligned with Keep a Changelog semantics and the scope of each change.
  • Eval chore: initialize marketplace scaffold #1 update is accurate. Updated expected_output correctly captures the new prerequisite-fail path: STOP + gap report.

Summary: One moderate finding (live-branch URL) worth resolving before merge; two low findings that would improve internal consistency. The substance of the PR is solid and the latest commit introduces no new concerns.

@claude

claude Bot commented Jul 21, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 2m 4s —— View job


Security Review — PR #845

Tasks

  • Load tools
  • Fetch PR diff / read changed files
  • Analyze for security vulnerabilities
  • Post findings

Scope

16 files changed — all markdown skill documents, plugin.json semver bumps, PLAN/design docs, and CHANGELOG entries. No GitHub Actions workflows, no executable code, no dependency pins, no secrets handling.

The latest commit in this sync (312f95d) touches only docs/topics/boris-video-absorption/PLAN.md (marks phases 6–7 as DONE). No new surface introduced; covered in the delta note below.


Finding 1 — Unpinned live-branch URL in AI-executable skill instructions

Severity: IMPORTANT
Confidence: CONFIRMED

Two files added by this PR embed a github.com/ghraw link pointing at the live main branch:

https://github.com/ghraw/melodic-software/claude-code-plugins/main/docs/conventions/consumer-config-layering/README.md

This is not merely a human-readable link. In an AI skill document, a complete github.com/ghraw URL with an inline description ("see the consumer-config layering contract") is an actionable instruction — an agent executing Step 2 of the skill may issue a WebFetch against it. Three concrete risks follow:

  1. Version decoupling: An agent running testing@0.3.0 fetches whatever main holds at execution time, not the layering contract that was current when 0.3.0 shipped. A breaking change to the layering contract on main takes immediate effect across all deployed plugin versions with no semver signal to consumers.

  2. Live prompt injection surface: Injected content served at this URL would enter the agent's instruction context during execution. Because the URL resolves within the same repository, exploitation requires write access to this repo's main branch — it is not a remote third-party trust risk. But branch protection gaps, stolen PATs, or compromised workflow write-back could make main writable in ways the plugin's consumers cannot audit at install time.

  3. Host coupling: If the repository is renamed or migrated, the link breaks silently for all consumers running the cached plugin version, which has no mechanism to detect the breakage.

The convention doc is not inside the plugin, so a relative path cannot reach it. Replacing the hyperlink with prose — naming the path docs/conventions/consumer-config-layering/README.md in the plugin marketplace — gives the reader and the AI the same navigational signal without creating a live fetch surface.

Fix this →


Finding 2 — New consumer config surface is an AI-readable prompt-injection vector

Severity: SUGGESTION
Confidence: PLAUSIBLE

plugins/testing/skills/run-e2e/SKILL.md Step 2 instructs the agent to read the full text of .claude/testing/e2e.md and merge it into the run's instruction context. A consumer repo hosting a maliciously crafted config file — containing adversarial prose alongside valid recording / browser_mode key-value lines — could influence the agent beyond the intended keys. Example: a file that sets recording: off but appends text that re-scopes the agent's next steps.

Exploitation requires write access to a consumer repo's .claude/ directory (insider or CI pipeline threat model — not remote). The closed scalar values (video|gif|off, headed|headless) and short key-set limit the viable injection window.

This risk is inherent to the AI-readable config-surface architecture this marketplace uses, not unique to this PR. It is noted here because this PR adds a new surface, expanding the attack area. No code change is required for this finding; teams operating in high-trust-boundary environments may wish to validate consumer config files in CI before use.


No other security issues found

  • Pre-computed context blocks (! backtick shell commands in SKILL.md lines 11–12) — git branch --show-current and git status --porcelain — are read-only git introspection with no user-controlled input. Pre-existing, unmodified by this PR. No injection risk.
  • GitHub Actions — no .github/workflows/ files changed.
  • Secrets / credentials — none referenced or handled anywhere in the diff.
  • Supply chain — version bumps are plugin.json version string fields only; no action refs, no package pins altered.
  • CLAUDE_PLUGIN_ROOT / CLAUDE_PLUGIN_DATA isolation — new context files at ${CLAUDE_PLUGIN_ROOT}/skills/run-e2e/context/; no ../ reach-outs observed.
  • routines.md instruction-provenance clause — adds a normative text section; no execution surface, URL, shell command, or config key introduced.

Delta commit — docs(topics): mark phases 6-7 done

312f95d modifies only docs/topics/boris-video-absorption/PLAN.md (two lines marking phases 6–7 DONE). No new execution surface, URL, shell command, or config key. No new security findings.


Summary: Finding 1 is the material item — the unpinned live-branch URL in AI-executable skill instructions is worth resolving before merge. Finding 2 is an inherent architectural property of the config-surface model, noted for awareness.

@kyle-sexton
kyle-sexton merged commit ed2c8d4 into main Jul 21, 2026
23 checks passed
@kyle-sexton
kyle-sexton deleted the feat/boris-video-absorption branch July 21, 2026 10:17
kyle-sexton added a commit that referenced this pull request Jul 21, 2026
…y trial (#871)

## What

Flips the last open phase tag in
`docs/topics/boris-video-absorption/PLAN.md` — Phase 8 (post-merge
trial) — to `[DONE]`, with a short trial record.

## Trial summary (medley, 2026-07-21, local-only)

- Config surface resolved with per-key provenance (local overlay
`.claude/testing/e2e.local.md`; team + user-global layers absent)
- `recording: video` honored — WebM artifact on disk (gitignored in
consumer repo)
- Induced prereq failure (run before orchestrator start) → hard STOP +
structured verification-environment gap report
- Evidence table carried session artifacts (recording path, playwright
session ID, transcript pointers)
- Bonus: explicit "run this headed" prompt beat the overlay's
`browser_mode: headless`
- Operator reviewed the recording

Docs-only; no plugin content changes.

## Related

No linked issue. Related: #845 (phases 1–7 of the same PLAN, merged).

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant