feat: marketplace & local plugins experiment toggle by Groenbech96 · Pull Request #736 · microsoft/BC-Bench

Magnus Hartvig Grønbech (Groenbech96) · 2026-07-07T07:02:43Z

Implements the marketplace & local plugins experiment toggle designed in #733. This branch includes the design spec + implementation plan (from #733) plus the full implementation.

What

A config-driven plugins experiment toggle (peer of instructions/skills/agents/mcp) that installs agent plugins into the running CLI so their skills are native during a run — measured like the other experiment levers.

Sources: marketplace (repo + pinned commit) or local (path); per-entry enabled; multiple at once.
Activation via the CLIs' real commands (plugin marketplace add + plugin install) — config injection (extraKnownMarketplaces/enabledPlugins) is trust-dialog-gated and ignored in headless mode, so it cannot be used (verified + corroborated by an independent write-up).
Commit pinning: the marketplace is cloned at its commit into <repo>/.bcbench/plugins/.
Concurrency-safe: each entry installs into a fresh per-entry isolated config home (COPILOT_HOME / CLAUDE_CONFIG_DIR under .bcbench/), applied to the plugin commands and the agent launch — so the max-parallel: 64 matrix on the shared self-hosted runner never shares the user-scope plugin store. No teardown; nothing touches the developer's real config.
Recording: ExperimentConfiguration.plugins: list[str] ("name@commit" / "name@local"), mirroring mcp_servers.
AL-LSP untouched.

How it was built

Executed the plan task-by-task (subagent-driven, fresh agent per task + per-task review + final whole-branch review):

ExperimentConfiguration.plugins field
clone_at_commit git helper (pinned shallow clone)
plugin_operations.py materialize + marketplace-name helpers
setup_plugins_from_config (per-entry isolated home install)
Copilot runner wiring
Claude runner wiring
config.yaml + EXPERIMENT.md + mock scenario
Final-review hardening (config-key validation, clean partial home on all failures, dedupe marketplace clone)

Verification

Full plugin lifecycle verified live on both CLIs (non-interactive, offline): marketplace add + install into a fresh COPILOT_HOME/CLAUDE_CONFIG_DIR, skill loads, real copilot -p session runs (auth via env token, no onboarding), real config untouched.
Full test suite: 576 passed, 1 skipped; ruff clean.

Notes / follow-up

A fresh CLAUDE_CONFIG_DIR authenticated headless session (via ANTHROPIC_API_KEY) is the one residual runtime check to confirm in CI (Copilot's is verified); noted in the spec's open questions.

…gins experiment toggle Adds a design spec and a task-by-task implementation plan for a new experiment toggle that installs agent plugins (a marketplace pinned to a git commit, or a local marketplace) into the running CLI via its own 'plugin marketplace add' + 'plugin install' commands, records them in ExperimentConfiguration, and removes them after the run. Docs only; no code changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…iew) Addresses review: execution-based categories run entries as a parallel matrix (max-parallel 64) on a shared self-hosted runner, and the CLI plugin store is user-scope/global, so concurrent installs/teardowns would race. Fix: install into a fresh per-entry config home (COPILOT_HOME / CLAUDE_CONFIG_DIR under .bcbench), applied to both the plugin commands and the agent launch. This removes teardown entirely (isolated home discarded with the checkout) and moots the teardown_plugins signature comment. Verified live on both CLIs (fresh home: install + skill load + headless session, real config untouched). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…r-entry home) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…es, dedupe marketplace clone Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Move the terminal AgentError out of 'case _' to after the match block in _materialize and _home_env_var, matching the existing pattern in types.py (EvaluationCategory). Behavior unchanged; clears the github-code-quality mixed-returns advisory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Per PR #736 review: move the shutil.which(...) resolution + not-found check to the top of run_copilot_agent / run_claude_code (before the 'Running...' log and all setup work), so a missing CLI fails fast. Behavior otherwise unchanged; the resolved binary is still passed to setup_plugins_from_config. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

CI's pre-commit 'ty check' hook failed: add_call[1]/install_call[1] were typed dict | None (kwargs.get('env')) and thus not subscriptable. Record env as a non-optional dict (kwargs.get('env') or {}). ty now exits 0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add a self-contained example marketplace (bcbench-example-marketplace + bcbench-example-plugin bundling one no-op skill) under instructions/microsoft-BCApps/plugins/bcbench-example/, referenced by a disabled local entry in config.yaml. Flip enabled: true to smoke-test that a plugin installs into the per-entry isolated home and its skill loads (verified live: marketplace add + install + skill list all succeed). Guarded by a test tying config.yaml -> example dir -> marketplace.json -> plugin.json so the example can't silently rot. EXPERIMENT.md documents it and the built-in marketplace-name collision caveat. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…(revert before merge) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…, install) Adds INFO logging around plugin setup so CI logs clearly show each step — 'Installing N plugin marketplace(s) into isolated COPILOT_HOME=...', 'Cloning plugin marketplace repo@commit' / 'Copying local plugin marketplace ...', 'Registering marketplace ... from ...', and 'Installing/Installed plugin ...'. Previously only the post-install line (and clone_at_commit's clone log) showed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Local `plugins` entries resolved their `path` under instructions/<owner>-<repo>/, which broke for dataset entries whose repo (e.g. microsoftInternal/NAV) differs from where the example was stored. Resolve `local` sources from a fixed, repo-agnostic base (src/bcbench/agent/shared/plugins/) instead, and move the bundled bcbench-example there. Update tests, spec, plan, EXPERIMENT.md, and the config.yaml comments/path accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The `marketplace` plugin source now accepts an optional `path` (marketplace root within the clone), mirroring `local`, so repos that host a marketplace in a subdirectory are supported. Enable a self-contained marketplace example that clones this repo at a pinned commit and points `path` at the bundled example, so an evaluation run visibly logs the plugin clone under bcbench.operations.git_operations (the local copy path logged no clone). The local example is kept as a disabled, test-validated template. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds a real-world marketplace plugin example (obra/superpowers, marketplace "superpowers-dev", plugin "superpowers") pinned to v6.1.1, enabled for a smoke run so the agent can discover and invoke a plugin skill (e.g. brainstorming) via the `skill` tool in headless -p mode. The prior bundled examples are kept as disabled, test-validated templates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ppet Installing a plugin makes its skills available but not used: on well-specified tasks the agent works directly and invokes nothing. Add an "Encouraging plugin usage" section to EXPERIMENT.md (MCP/hooks are automatic; skills are discretionary and need a nudge via the instructions lever or category prompt), and ship a ready-to-use, subtle nudge snippet (instructions/skill-usage-nudge.md) that experimenters can append to a repo AGENTS.md and toggle on with instructions.enabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Drop the obra/superpowers marketplace example (it was only enabled to demo skill usage). The two bundled examples (local bcbench-example and the BC-Bench self-clone marketplace) remain as disabled templates, so nothing installs by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Remove the blank line before setup_plugins_from_config in both runners so it groups with the other setup_* calls. - Drop the "what we're not doing" (extraKnownMarketplaces) explanation from the plugin_operations module docstring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Per review: keep the local bcbench-example and the obra/superpowers marketplace example (both disabled), remove the BC-Bench self-clone entry, and cut the plugins comment block down to the essentials to match the rest of the config. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Reduce the "Encouraging plugin usage" section to the single reliable lever (custom instructions) and inline the subtle nudge as a copy-paste block instead of a standalone file under instructions/ (which is not an established location for loose markdown). Remove the skill-usage-nudge.md file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

These docs/superpowers/ spec and plan files were superpowers-workflow artifacts and are not part of this repo's docs. The design rationale is captured in EXPERIMENT.md and the code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…l-plugins

…dd failure) An earlier commit failed to stage EXPERIMENT.md because a sibling deleted-file pathspec errored, so the "point 1 only" trim + inlined nudge never landed. Commit it now, removing the broken link to the deleted snippet file and the extra points. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

bcbench and others added 12 commits July 6, 2026 09:44

docs: clarify uninstall/remove are verified-reversible but unused (pe…

ae38816

…r-entry home) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

feat: record installed plugins in ExperimentConfiguration

c6faee2

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

feat: add clone_at_commit git helper for pinned marketplace clones

4d02f61

feat: add plugin materialize + marketplace-name helpers

090c28d

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

feat: install plugins into a per-entry isolated CLI config home

68a7854

feat: install plugins into a per-entry home in the Copilot runner

459c4a8

feat: install plugins into a per-entry home in the Claude runner

323db0f

docs+config: document plugins toggle, add config block and mock scenario

7b6df39

style: fix ruff E402/PT019 in test_plugin_operations

545b4d1

harden: validate plugin config keys, clean partial home on all failur…

5588d76

…es, dedupe marketplace clone Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-code-quality Bot found potential problems Jul 7, 2026

View reviewed changes

Comment thread src/bcbench/operations/plugin_operations.py Fixed

Comment thread src/bcbench/operations/plugin_operations.py Fixed

Magnus Hartvig Grønbech (Groenbech96) commented Jul 7, 2026

View reviewed changes

Comment thread src/bcbench/agent/claude/agent.py Outdated

Comment thread src/bcbench/agent/copilot/agent.py Outdated

Comment thread src/bcbench/operations/__init__.py

bcbench and others added 10 commits July 7, 2026 09:43

test: enable bundled example plugin to smoke-test copilot-evaluation …

1e9fa91

…(revert before merge) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Magnus Hartvig Grønbech (Groenbech96) commented Jul 8, 2026

View reviewed changes

Magnus Hartvig Grønbech and others added 4 commits July 8, 2026 09:06

Magnus Hartvig Grønbech and others added 2 commits July 8, 2026 09:36

Merge remote-tracking branch 'origin/main' into feat/marketplace-loca…

d98e733

…l-plugins

Magnus Hartvig Grønbech (Groenbech96) mentioned this pull request Jul 8, 2026

Experiment: code-review with BCQuality plugin #737

Closed

Magnus Hartvig Grønbech (Groenbech96) enabled auto-merge (squash) July 8, 2026 08:17

Wenjie Fan (gggdttt) approved these changes Jul 8, 2026

View reviewed changes

Magnus Hartvig Grønbech (Groenbech96) merged commit f0c7c5d into main Jul 8, 2026
13 checks passed

Magnus Hartvig Grønbech (Groenbech96) deleted the feat/marketplace-local-plugins branch July 8, 2026 11:46

Copilot AI mentioned this pull request Aug 6, 2026

chore: bump version to 0.8.0 #775

Merged

Navigation Menu

Uh oh!

feat: marketplace & local plugins experiment toggle - #736

feat: marketplace & local plugins experiment toggle#736
Magnus Hartvig Grønbech (Groenbech96) merged 29 commits into
mainfrom
feat/marketplace-local-plugins

Magnus Hartvig Grønbech (Groenbech96) commented Jul 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Magnus Hartvig Grønbech (Groenbech96) commented Jul 7, 2026

What

How it was built

Verification

Notes / follow-up

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants