Skip to content

Add an mcp level to the agent benchmark #474

Description

@e54-bot

Problem

There is no measurement of whether native Wright tools improve on shell + JSON for task success and search/read/tool-call efficiency. See ADR-0020 (Proposed in the ADR PR for #470).

Scope

Add mcp as a wright level beside bin in the agent benchmark condition grid (<wright>/<knowledge>/<network>), with scenario, prompt, grader, model, and trial identity unchanged:

  • the harness describes the level to the adapter, and the reference adapter registers the wright serve --transport mcp server for the agent;
  • the Wright trace records MCP tool calls so wrightUse, friction, and the report cover them;
  • the report compares mcp with bin on usable/passed rate with Wilson intervals and on search/read/tool-call counts, turns, and tokens; tool-schema context counts toward the mcp cost;
  • docs/agent-benchmark.md documents the level.

Non-goals

Acceptance criteria

  • agent_bench.py validate and the existing benchmark unit tests pass unchanged.
  • A run with level mcp produces result.json with MCP calls counted in wrightUse, and a canary fails the run as invalid when wright is reachable under level none, as today.
  • report shows mcp versus bin paired on the same scenario, agent, and trial, with the metrics listed above.
  • Tool-schema tokens are included in mcp input-token accounting, verified on a fixture usage log.

Verification

Cases in benchmarks/agent/test_agent_bench.py; one real run recorded in the PR as evidence that the level works end to end (not as a product claim).

Dependencies / ownership

Owner: Wright (benchmarks/agent). Parent: #470. Depends on the MCP adapter Issue. Orthogonal to #466.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions