AgentProbe

A ReAct agent observatory built to instrument the gap between what an agent decides and what actually happens in production.

Most agent demos work. Most agent deployments don't. AgentProbe is a from-scratch ReAct loop — parser, dispatcher, failure detector, streaming layer, all auditable — instrumented with an 8-category failure taxonomy and a decision graph view that turns every run into reproducible diagnostic data.

Above: a single agent run rendered as a decision graph (live capture from production). The red WEATHER_FORECAST action marks the deterministic detection of hallucinated_tool — the agent invoked a tool the registry didn't have, the orchestrator surfaced an [ERROR] observation, the agent recovered with web_search, then drifted into a goal_drift final answer about Lyon's weather instead of its population. Both failures appear as red badges on the run header; both are reproducible from the seeded demo run at /runs/demo-fail-001.

Live demo: agent-probe-one.vercel.app/runs/demo-fail-001 · demo-happy-001 · API health

Why this exists

The agent industry is converging on a single uncomfortable truth: the bug isn't in the model — it's in the gap between the model's intent and the action that gets executed. Without something inspecting every reasoning step, every tool call, and every failure mode, you're shipping confidence with no grounding.

AgentProbe is the diagnostic side of that problem. It does three things competing tools usually don't:

No framework lock-in. The ReAct loop is implemented from scratch (~500 LOC) — every line of the reasoning path is auditable, not buried in a LangChain abstraction.
A real failure taxonomy. 8 categories, each with deterministic detection rules. hallucinated_tool is a tool name lookup, not a vibe. goal_drift is keyword overlap, not an LLM judge.
Decision-graph visualization. Every run renders as a directed graph of reasoning nodes — Thought → Action → Observation cycles, with failures inline on the failing edge. The same shape any explainability or audit surface needs.

Built to be the kind of tool a serious agent platform team would either build internally or buy.

Features

Feature	Description
Decision Graph view	Every run renders as a directed graph of reasoning nodes; failures appear inline on the failing edge
Real-time Playground	Type a query, watch the agent reason step-by-step via SSE streaming
8-Type Failure Taxonomy	Deterministic classification: hallucinated tools, malformed actions, context overflow, goal drift, etc.
Multi-Model Benchmarking	50+ test cases across 5 categories with composite scoring (answer + tools + efficiency + reliability)
Cross-Model Compare	Side-by-side dual-trace execution on the same query
5 LLM Providers	Groq, Ollama, OpenAI, Anthropic, Google — with dynamic availability detection
Run Replay	Replay any persisted run as an SSE stream, identical to the original execution
Analytics Dashboard	Failure distributions, model performance heatmaps, KPI overview cards
Custom Tools	Build HTTP or static tools via the UI — agent uses them in real-time
Prompt Engineering	Create, save, and A/B test custom system prompts
Agent Memory	Persistent key-value memory across runs via save/recall tools
Auth + multi-tenant	JWT + API keys, per-user data scoping, RBAC-ready row filtering
Export	CSV and PDF export for benchmark results and run traces

Decision graph view

Every persisted run can be viewed as a directed graph of reasoning steps. Each row is one ReAct cycle (Thought → Action → Observation), with the chosen tool labelled on the action node and any failure surfaced inline as a red badge plus dashed edge. Click a node to inspect the raw content, latency, and token cost — the same shape any explainability or audit surface needs.

Open any run at /runs/<id> and toggle Graph / List. The graph view is the default.

Screenshots

All captures are pulled live from the production deploy at agent-probe-one.vercel.app — no mockups, no styled stills.

_{Hallucinated tool detection. Click the red WEATHER_FORECAST action node — the side panel surfaces the deterministic detection rule (tool not in registry), the offending arguments, and the step index.}	_{Goal drift on the final answer. Click the red DECISION node — the agent answered about Lyon's weather, not its population. Detected by keyword overlap, surfaced as a goal_drift badge on the run header.}
_{List view. Same run, flat chronological. Useful when you want to read the actual [ERROR] observation text without spatial reasoning about graph layout.}	_{Analytics dashboard. Aggregate failure distribution, cross-model comparison (avg steps, avg tokens, success rate), and a per-model failure breakdown — all driven by SQL aggregates over the persisted runs.}

Captures are reproducible: node scripts/capture-screenshots.mjs re-runs the full set against the live deploy.

Architecture

graph TB
    subgraph Frontend["Next.js 16 Frontend"]
        P[Playground]
        R[Run History]
        B[Benchmarks]
        A[Analytics]
        C[Model Compare]
        PR[Prompts]
    end

    subgraph Backend["FastAPI Backend"]
        subgraph Domain["Domain Layer"]
            E[Entities]
            PO[Ports / Interfaces]
        end
        subgraph Application["Application Layer"]
            O[AgentOrchestrator]
            EH[EvalHarness]
            AN[AnalyticsService]
            SE[ScoringEngine]
            AU[AuthService]
            EX[ExportService]
        end
        subgraph Infrastructure["Infrastructure Layer"]
            PRV[LLM Providers]
            DB[(PostgreSQL)]
            T[Tool Registry]
            MW[Middleware]
        end
    end

    Frontend -->|SSE / REST| Backend
    O --> PRV
    O --> T
    O --> DB
    EH --> O
    AN --> DB
    AU --> DB

    style Frontend fill:#1a1a2e,stroke:#4fc3f7,color:#e0e0e0
    style Domain fill:#0d1b2a,stroke:#81c784,color:#e0e0e0
    style Application fill:#0d1b2a,stroke:#ffb74d,color:#e0e0e0
    style Infrastructure fill:#0d1b2a,stroke:#ef5350,color:#e0e0e0

Clean Architecture Layers

graph LR
    D[Domain] -->|depends on nothing| D
    A[Application] -->|depends on| D
    I[Infrastructure] -->|depends on| A
    I -->|depends on| D

    style D fill:#1b5e20,stroke:#81c784,color:#fff
    style A fill:#e65100,stroke:#ffb74d,color:#fff
    style I fill:#b71c1c,stroke:#ef5350,color:#fff

Domain — Entities (AgentRun, AgentStep, User, CustomTool, MemoryEntry), enums (FailureType, StepType), port interfaces (zero external dependencies)
Application — AgentOrchestrator (ReAct loop), EvalHarness, ScoringEngine, AnalyticsService, AuthService, ExportService, Pydantic schemas
Infrastructure — 5 LLM providers, SQLAlchemy persistence (9 tables), tool registry, FastAPI routes, middleware (rate limiter, auth, request validator)

ReAct Agent Loop

flowchart TD
    Start([User Query]) --> SYS[Emit System Step]
    SYS --> LOOP{Step < Max?}
    LOOP -->|Yes| CTX{Context OK?}
    CTX -->|Overflow| ERR1[Context Overflow Error]
    CTX -->|OK| LLM[Call LLM Provider]
    LLM --> PARSE[Parse Output]
    PARSE -->|Final Answer| FINAL[Persist Run + Return]
    PARSE -->|Action| CHECK{Tool Exists?}
    CHECK -->|No| FAIL1[Hallucinated Tool]
    CHECK -->|Yes| DUP{Repeated?}
    DUP -->|Yes| FAIL2[Repeated Action]
    DUP -->|No| EXEC[Execute Tool]
    EXEC --> OBS[Observation]
    OBS --> LOOP
    PARSE -->|Malformed| FAIL3[Malformed Action]
    FAIL1 --> LOOP
    FAIL2 --> LOOP
    FAIL3 --> LOOP
    LOOP -->|No| TIMEOUT[Max Steps Exceeded]

    style Start fill:#4fc3f7,stroke:#4fc3f7,color:#000
    style FINAL fill:#81c784,stroke:#81c784,color:#000
    style FAIL1 fill:#ef5350,stroke:#ef5350,color:#fff
    style FAIL2 fill:#ef5350,stroke:#ef5350,color:#fff
    style FAIL3 fill:#ef5350,stroke:#ef5350,color:#fff
    style TIMEOUT fill:#ef5350,stroke:#ef5350,color:#fff
    style ERR1 fill:#ef5350,stroke:#ef5350,color:#fff

Every failure is recorded with its type, step index, and context — enabling the aggregate analytics that power the dashboard.

Failure Taxonomy

The core differentiator. Every run is annotated with exactly which failure modes occurred:

mindmap
  root((Failure Types))
    Tool Failures
      hallucinated_tool
      tool_execution_error
    Parse Failures
      malformed_action
      empty_response
    Loop Failures
      max_steps_exceeded
      context_overflow
      repeated_action
    Quality Failures
      goal_drift

Failure	What Happened	How It's Detected
`hallucinated_tool`	LLM invented a tool name	Tool name not in registry
`malformed_action`	Can't parse Action/Action Input	Regex parse failure
`tool_execution_error`	Tool threw an exception	`[ERROR]` prefix in observation
`max_steps_exceeded`	Agent never reached Final Answer	Step counter >= limit
`context_overflow`	Context window approaching limit	Character count check
`repeated_action`	Same tool + args called twice	Dedup check on recent steps
`empty_response`	LLM returned nothing	Empty string check
`goal_drift`	Final answer doesn't address query	Keyword overlap analysis

Tech Stack

graph LR
    subgraph Frontend
        NX[Next.js 16]
        RE[React 19]
        TS[TypeScript 5]
        TW[Tailwind CSS 4]
        SH[shadcn/ui]
        ZU[Zustand]
        TQ[TanStack Query]
        RC[Recharts]
        PW[Playwright]
    end

    subgraph Backend
        FA[FastAPI]
        PY[Python 3.10+]
        SA[SQLAlchemy]
        AL[Alembic]
        PD[Pydantic v2]
    end

    subgraph Providers
        GR[Groq]
        OL[Ollama]
        OA[OpenAI]
        AN[Anthropic]
        GO[Google Gemini]
    end

    subgraph Data
        PG[(PostgreSQL)]
        SQ[(SQLite dev)]
    end

    subgraph DevOps
        DK[Docker Compose]
        GA[GitHub Actions]
        VE[Vercel]
        RW[Railway]
    end

    style Frontend fill:#1a1a2e,stroke:#4fc3f7,color:#e0e0e0
    style Backend fill:#1a1a2e,stroke:#81c784,color:#e0e0e0
    style Providers fill:#1a1a2e,stroke:#ffb74d,color:#e0e0e0
    style Data fill:#1a1a2e,stroke:#ce93d8,color:#e0e0e0
    style DevOps fill:#1a1a2e,stroke:#ef5350,color:#e0e0e0

Quick Start

Docker (recommended)

# Clone and configure
git clone https://github.com/soneeee22000/AgentProbe.git
cd AgentProbe

# Set API keys
cat > .env << EOF
GROQ_API_KEY=your_key_here
TAVILY_API_KEY=your_key_here
EOF

# Launch full stack (PostgreSQL + Backend + Frontend)
docker-compose up --build

# Frontend: http://localhost:3000
# Backend:  http://localhost:8000
# API Docs: http://localhost:8000/docs

Demo data — see the decision graph immediately

After the stack is up, seed two illustrative runs (one success, one failure with hallucinated_tool + goal_drift) so the graph view has something interesting to show without a network call:

# From repo root, with the backend running
docker compose exec backend python -m scripts.seed_demo_runs

# Or against a local SQLite dev DB
cd backend && python -m scripts.seed_demo_runs

Then open:

http://localhost:3000/runs/demo-happy-001 — clean ReAct path (calculator → final answer)
http://localhost:3000/runs/demo-fail-001 — failure trace with two distinct failure modes inline on the graph

Manual Setup

Backend

cd backend
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env    # Add your API keys
mkdir workspace
uvicorn main:app --reload --port 8000

Frontend

cd frontend
npm install
npm run dev              # http://localhost:3000

API Keys

Provider	Free Tier	URL
Groq	Yes	https://console.groq.com
Tavily	Yes	https://tavily.com
Ollama	Local	https://ollama.com
OpenAI	Paid	https://platform.openai.com
Anthropic	Paid	https://console.anthropic.com
Google	Free	https://aistudio.google.com

API Reference

Agent Execution

Method	Endpoint	Description
`POST`	`/api/v1/run`	Start agent run (SSE stream)

Run Management

Method	Endpoint	Description
`GET`	`/api/v1/runs`	List runs (paginated, filterable)
`GET`	`/api/v1/runs/{id}`	Run detail with full step trace
`DELETE`	`/api/v1/runs/{id}`	Delete a run
`GET`	`/api/v1/runs/{id}/replay`	Replay run as SSE stream

Benchmarking

Method	Endpoint	Description
`GET`	`/api/v1/benchmarks/cases`	List benchmark test cases
`POST`	`/api/v1/benchmarks/cases`	Create custom test case
`POST`	`/api/v1/benchmarks/suites`	Start benchmark suite (SSE progress)
`GET`	`/api/v1/benchmarks/suites`	List completed suites
`GET`	`/api/v1/benchmarks/suites/{id}`	Suite detail with per-case results

Analytics & Providers

Method	Endpoint	Description
`GET`	`/api/v1/analytics/failures`	Failure type breakdown
`GET`	`/api/v1/analytics/models`	Cross-model performance stats
`GET`	`/api/v1/providers`	Available providers + models
`GET`	`/api/v1/tools`	Built-in tool list
`GET`	`/api/v1/health`	Health check with provider status

Custom Tools & Prompts

Method	Endpoint	Description
`POST`	`/api/v1/tools/custom`	Create custom tool
`GET`	`/api/v1/tools/custom`	List custom tools
`DELETE`	`/api/v1/tools/custom/{id}`	Delete custom tool
`POST`	`/api/v1/prompts`	Create prompt template
`GET`	`/api/v1/prompts`	List prompt templates
`PUT`	`/api/v1/prompts/{id}`	Update prompt template
`DELETE`	`/api/v1/prompts/{id}`	Delete prompt template

Auth & Export

Method	Endpoint	Description
`POST`	`/auth/register`	Register new user
`POST`	`/auth/login`	Login (returns JWT)
`POST`	`/auth/api-keys`	Generate API key
`GET`	`/auth/me`	Current user info
`GET`	`/api/v1/exports/runs/{id}/csv`	Export run as CSV
`GET`	`/api/v1/exports/benchmarks/{id}/csv`	Export benchmark as CSV
`GET`	`/api/v1/exports/benchmarks/{id}/pdf`	Export benchmark as PDF

Database Schema

erDiagram
    users ||--o{ api_keys : has
    users ||--o{ runs : owns
    users ||--o{ custom_tools : creates
    users ||--o{ prompt_templates : creates
    users ||--o{ memory_entries : stores
    runs ||--o{ steps : contains
    runs ||--o{ failures : records
    benchmark_suites ||--o{ benchmark_results : contains
    benchmark_cases ||--o{ benchmark_results : tested_by
    runs ||--o{ benchmark_results : produces

    users {
        string id PK
        string email UK
        string hashed_password
        datetime created_at
    }
    runs {
        string id PK
        string query
        string model_id
        string provider
        string status
        string final_answer
        int total_tokens
        float duration_ms
        bool succeeded
        string user_id FK
    }
    steps {
        int id PK
        string run_id FK
        int step_index
        string step_type
        string content
        string tool_name
        string failure_type
    }
    failures {
        int id PK
        string run_id FK
        int step_id FK
        string failure_type
        string context
    }
    benchmark_cases {
        string id PK
        string query
        string category
        string difficulty
    }
    benchmark_suites {
        string id PK
        string model_id
        string provider
        float success_rate
    }
    custom_tools {
        string id PK
        string user_id FK
        string name
        string tool_type
    }
    memory_entries {
        string id PK
        string user_id FK
        string key
        string value
    }

9 tables total. SQLite for local development, PostgreSQL for production. Alembic manages migrations.

Project Structure

agentprobe/
├── backend/
│   ├── src/agentprobe/
│   │   ├── domain/                  # Zero-dependency core
│   │   │   ├── entities/            # AgentRun, Step, User, CustomTool, Memory, Prompt
│   │   │   └── ports/               # ILLMProvider, IRunRepository, IUserRepository, ...
│   │   ├── application/             # Business logic
│   │   │   ├── services/            # Orchestrator, EvalHarness, Auth, Analytics, Export
│   │   │   └── schemas/             # Pydantic v2 request/response models
│   │   └── infrastructure/          # External integrations
│   │       ├── api/                 # FastAPI app, routes (10 modules), middleware (3)
│   │       ├── providers/           # Groq, Ollama, OpenAI, Anthropic, Google
│   │       ├── persistence/         # SQLAlchemy ORM (9 tables), 4 repositories
│   │       └── tools/               # calculator, web_search, think, read_file, memory, custom
│   ├── tests/                       # 81+ tests (unit, integration, API)
│   ├── alembic/                     # 3 migration files
│   └── pyproject.toml
│
├── frontend/
│   ├── src/
│   │   ├── app/                     # 7 routes (/, /runs, /benchmarks, /analytics, /compare, /prompts, dynamic)
│   │   ├── components/              # 30+ components across 8 modules
│   │   ├── store/                   # Zustand (run-store, compare-store)
│   │   └── lib/                     # API client with SSE streaming
│   ├── e2e/                         # Playwright E2E tests (4 specs)
│   └── playwright.config.ts
│
├── .github/workflows/               # CI (lint, test, build, E2E) + Deploy
├── docker-compose.yml               # PostgreSQL + Backend + Frontend
├── CLAUDE.md                        # Project instructions
└── PRD.md                           # Product requirements

Development

# Backend lint + test + type check
cd backend
ruff check src/ tests/
pytest --cov=src/agentprobe
mypy src/

# Frontend lint + build + E2E
cd frontend
npm run lint
npm run build
npm run e2e

Scoring Engine

Benchmark cases are scored with a weighted composite:

pie title Composite Score Weights
    "Answer Correctness" : 40
    "Tool Usage" : 20
    "Efficiency" : 20
    "Reliability" : 20

Answer Correctness (40%) — Keyword overlap between agent's answer and expected answer
Tool Usage (20%) — Did the agent use the expected tools?
Efficiency (20%) — Fewer steps = higher score
Reliability (20%) — No failures = full score; each failure type deducts proportionally

Roadmap

Phase 1 — Clean Architecture, Groq + Ollama, SSE streaming, 6 tables
Phase 2 — Benchmarking (50+ cases), analytics dashboard, composite scoring
Phase 3 — Multi-model compare, Docker Compose, 81 tests
Phase 4 — PostgreSQL + Alembic, rate limiting, request validation
Phase 5 — OpenAI, Anthropic, Google providers + dynamic discovery
Phase 6 — Auth (JWT + API keys), GitHub Actions CI/CD
Phase 7 — Custom tools, prompt engineering UI, agent memory
Phase 8 — Vercel + Railway deploy, Playwright E2E, CSV/PDF export

License

MIT

Navigation Menu

Repository files navigation

AgentProbe

Why this exists

Features

Decision graph view

Screenshots

Architecture

Clean Architecture Layers

ReAct Agent Loop

Failure Taxonomy

Tech Stack

Quick Start

Docker (recommended)

Demo data — see the decision graph immediately

Manual Setup

API Keys

API Reference

Agent Execution

Run Management

Benchmarking

Analytics & Providers

Custom Tools & Prompts

Auth & Export

Database Schema

Project Structure

Development

Scoring Engine

Roadmap

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
backend		backend
docs		docs
frontend		frontend
scripts		scripts
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
PRD.md		PRD.md
README.md		README.md
docker-compose.yml		docker-compose.yml

Latest commit

History

Folders and files

Repository files navigation

AgentProbe

Why this exists

Features

Decision graph view

Screenshots

Architecture

Clean Architecture Layers

ReAct Agent Loop

Failure Taxonomy

Tech Stack

Quick Start

Docker (recommended)

Demo data — see the decision graph immediately

Manual Setup

API Keys

API Reference

Agent Execution

Run Management

Benchmarking

Analytics & Providers

Custom Tools & Prompts

Auth & Export

Database Schema

Project Structure

Development

Scoring Engine

Roadmap

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages