Skip to content

Home

Jump to bottom
irparent edited this page Sep 3, 2026 · 2 revisions

Iris — stop shipping agents on vibes

Iris is an open-source MCP server that scores every agent run for quality, safety, and cost. 13 built-in rules catch PII leaks, prompt injection, hallucination signals, and cost overruns before your users do — deterministically, on your machine, with no LLM call.

Website | Playground | npm | Docs

Quick Start

A failure on screen in 60 seconds, no agent wiring:

npx @iris-eval/mcp-server --demo

Then add Iris to your agent's MCP config (Claude Desktop, Claude Code, Cursor, Windsurf, or any MCP-compatible client):

{
  "mcpServers": {
    "iris-eval": {
      "command": "npx",
      "args": ["@iris-eval/mcp-server", "--dashboard"]
    }
  }
}

The dashboard serves at http://localhost:6920 and lands on Failures — what failed, worst and newest first.

Documentation

The docs live in the repository, next to the code they describe:

  • README — install, the nine tools, how passed is decided, configuration
  • Architecture — how Iris works under the hood
  • Custom rules — the eight rule types, scoring, ReDoS protection
  • API reference — tool schemas and dashboard REST endpoints
  • HTTP ingest — POST /api/v1/traces, capture with no model in the loop
  • LLM-as-judge — semantic scoring, bring your own key, cost-capped
  • OpenTelemetry export — OTLP/HTTP to any collector
  • Roadmap — what is shipped and what is being considered
  • Contributing — how to contribute

Why Agent Eval?

Traditional monitoring tells you if your agent ran. Iris tells you how well it performed.

What monitoring catches What Iris catches
Agent crashed Agent leaked a credit card number
Latency > 5s Agent fabricated a figure the source never stated
Error rate spike Agent repeated an injection attack in output
Cost per request Cost 4.7x over budget for a simple query

Architecture Overview

Iris is an MCP server. The agent discovers its tools and calls them — log_trace to record a run, evaluate_output to score it — and the results land in a local SQLite database and the dashboard. Code that must record a run whether or not the model chooses to can post it to the HTTP ingest endpoint instead.

MCP Client (Claude, Cursor, ...)
    ↓ tool calls: log_trace, evaluate_output, get_traces, ...
Iris MCP Server
    ├── 13 Eval Rules (PII, injection, hallucination signals, cost, ...)
    ├── SQLite Storage (traces, evals, metrics) — ~/.iris/iris.db
    └── Web Dashboard (localhost:6920, --dashboard) + POST /api/v1/traces

The 13 Eval Rules

Category Rule What it catches
Completeness non_empty_output Empty or whitespace-only output
Completeness min_output_length Truncated or trivially short answers
Completeness sentence_count One-word answers to real questions
Completeness expected_coverage Expected terms missing from the response
Relevance keyword_overlap Output vocabulary that ignores the input
Relevance topic_consistency Output drifting from the prompt topic
Safety no_pii Credit cards, SSNs, emails, phone numbers, API keys and other credentials — critical: a failure fails the whole eval
Safety no_injection_patterns Prompt injection repeated in output — critical
Safety no_blocklist_words Banned terms, competitor names — critical
Safety no_stub_output TODO, [INSERT, and other placeholder markers
Safety no_hallucination_markers Claims contradicted by, or absent from, the provided input
Cost cost_under_threshold Query cost over budget
Cost token_efficiency Token usage out of proportion to the output

passed is true only when the weighted score clears the threshold (default 0.7) and no critical rule failed. Pass eval_type: "safety" to run the safety bundle — the default bundle is completeness.

License

MIT License. Copyright (c) 2026 Ian Parent.

Clone this wiki locally