Skip to content

tracking: orchestrator eval patterns — multi-agent, evidence verification, session replay, quality gates #337

Description

@christso

Summary

Tracking issue for AgentV gaps identified from researching two orchestrator projects:

What AgentV already covers

After investigation, AgentV already implements several features these orchestrators built custom:

  • Workspace lifecyclebefore_all, before_each, after_all, after_each with script execution and context passing
  • Quality gatingrequired on evaluators, required_min_score on rubrics, score ranges
  • Multi-target matrix — per-test targets array, parallel workers, batch providers
  • Trials/statistical robustnesspass_at_k, mean, confidence_interval strategies
  • Cross-run comparisonagentv compare with delta/win/loss/tie
  • Agent investigationagent_judge evaluator with filesystem tools for workspace audit
  • Tool trajectory — validate tool call sequences with ordering modes
  • Execution metrics — duration, cost, tokens, tool calls in trace
  • Transcript import — import and replay pre-recorded Claude/Codex/Copilot sessions via TranscriptProvider
  • N-way trend analysisagentv trend command with linear regression, slope thresholds, --fail-on-degrading
  • Severity expressivenessrequired + weight already express error/warning/info semantics
  • Custom evaluators.agentv/graders/ and .agentv/assertions/ directories for user-defined evaluator types without core changes

Remaining gap

# Issue Priority What's missing
1 #331 Multi-agent eval with dependencies P1 Test dependency DAG, wave scheduling, inter-test result passing — requires core orchestrator changes

Closed / resolved without core changes

# Issue Resolution
#332 Transcript evidence & claim verification Not needed as core — full agent output already flows to evaluators; achievable with llm-grader prompt patterns or .agentv/graders/ code-grader scripts
#333 Session recording & replay Not needed as core — transcript import handles replay; live recording is a thin wrapper or after_each hook; ralph-orchestrator cassettes can be imported
#334 Severity levels & auto-remediation YAGNIrequired: true = error, required: false + weight: 0 = info; auto-remediation belongs in CI/CD, not eval framework
#335 Iteration tracking & termination taxonomy Not needed as coreagentv trend exists; iteration counts and termination reasons are agent-specific metadata extractable via code-grader
#336 Workspace lifecycle Already implemented via beforeAll/beforeEach/afterAll/afterEach

Why only #331 needs core changes

The other features can all be implemented with AgentV's existing extension points:

  • Custom evaluators (.agentv/graders/, .agentv/assertions/) receive full EvaluationContext including output messages, trace, token usage, file changes, workspace path, and metadata
  • LLM grader can run arbitrary verification prompts against agent output (claim verification, drift detection)
  • Code grader can parse transcripts, extract iteration counts, classify termination reasons
  • Hooks (before_each, after_each) can record/replay session data
  • Transcript import already replays pre-recorded sessions through the full evaluation pipeline

#331 is the exception because test dependency ordering and inter-test result passing are orchestrator-internal concerns that can't be accessed from evaluators or hooks.

Research artifacts

  • research/findings/ralph-orchestrator/README.md — eval-focused analysis
  • research/findings/copilot-swarm-orchestrator/README.md — full analysis with 8 gap areas (revised to 1)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions