Iris scores every agent run for quality, safety, and cost — on your machine, with no SDK and no account. Most agent projects check quality by running a few remembered prompts and eyeballing the output. Iris replaces that with numbers you can audit: your agent's runs land in a SQLite database on your disk, 20 built-in rules score them deterministically — PII, prompt injection, hallucination markers, cost thresholds, and the agent's own tool calls — free, with no LLM calls, and an optional LLM judge with a hard per-eval cost cap handles the semantic questions. Every rule is inspectable and editable, because a judge you can't audit is just vibes with a number on it. MIT licensed, no telemetry. Nothing leaves your machine unless you set IRIS_OTEL_ENDPOINT, which exports traces to the collector you name, or enable the LLM judge with your own key.
Requires Node.js 20 or later. Check with node --version.
No agent wiring, no config — one command:
npx @iris-eval/mcp-server --demoThis seeds a demo database — a handful of small agents with a week of runs — and serves the dashboard against it at http://localhost:6920 (your browser opens automatically on first run). The dashboard lands on Failures: what failed, worst and newest first. Worth clicking into — a PII leak caught by the safety rules, a flagged prompt-injection attempt, and a failed LLM-judge score with its rationale.
Demo data lives in its own database (demo.db in your Iris home directory — ~/.iris on macOS/Linux, %USERPROFILE%\.iris on Windows) and never mixes with your real traces. Remove all of it with one command:
npx @iris-eval/mcp-server --demo-clearFirst, prove the install works on this machine — it runs offline and opens nothing of yours:
npx @iris-eval/mcp-server --self-test # exit 0 = healthyThen add Iris to your MCP config. Works with Claude Desktop, Claude Code, Cursor, Windsurf, Continue, VS Code, Cline, Zed, Codex CLI, Gemini CLI — and any other MCP-compatible agent. One block, dashboard included:
{
"mcpServers": {
"iris-eval": {
"command": "npx",
"args": ["@iris-eval/mcp-server", "--dashboard"]
}
}
}Your client lists Iris's twelve tools on connect, and the dashboard serves at http://localhost:6920. Now paste this to your agent:
Log that last task to Iris and evaluate the output.
The trace lands on the dashboard with its scores. Prefer the MCP server headless? Drop --dashboard from the args — you can open the same dashboard any time with npx @iris-eval/mcp-server --dashboard.
One thing worth knowing up front: MCP tools are called when the model decides to call them. Iris doesn't intercept your agent, so traces are logged when your agent asks it to log them — either because you told it to, or because your code calls the tools directly. Ask your agent to "log this to Iris and evaluate it" and it will. If you want capture that doesn't depend on the model choosing, POST /api/v1/traces does exactly that — your code sends the trace over plain HTTP, no model in the loop (see docs/http-ingest.md). The CLI and SDKs on the roadmap will be thin clients over the same endpoint.
The ingest endpoint lives on the dashboard port — 6920 by default, not the MCP transport port — and it exists only while the dashboard is running. Pass --dashboard (or set IRIS_DASHBOARD=true); --transport http on its own does not start it, and a request to the transport port returns 404. With the dashboard up, anything that can send an HTTP request can log a trace — and optionally run the deterministic evals in the same request. GET /api/v1/capabilities on the same port says what this server can judge, what each rule needs, the judge state with the steps that enable it, and the limits — the same object the MCP resource iris://capabilities serves — so an HTTP caller has the frame an MCP client gets at initialize:
curl -s -X POST "http://127.0.0.1:6920/api/v1/traces" \
-H "Content-Type: application/json" \
-d '{
"agent_name": "support-bot",
"input": "What is the refund policy?",
"output": "Refunds are available within 30 days of purchase.",
"evaluate": true,
"eval_type": "safety"
}'Returns 201 with the stored trace_id and the evaluation result (in --demo mode the endpoint refuses writes with 403, so demo data never mixes with yours). The endpoint accepts the same body as the log_trace tool and sits behind the same middleware stack as the rest of the dashboard: loopback bind and the DNS-rebinding guard by default, plus Bearer auth when you set one. Two plain facts about it: it accepts unauthenticated writes unless Iris was started with --api-key (or IRIS_API_KEY) — the loopback bind is what keeps it to your machine by default, so set a key before binding beyond loopback; and what it stores is verbatim — input and output land in iris.db exactly as sent, including any text no_pii goes on to flag. Full contract, field reference, and error semantics: docs/http-ingest.md.
/plugin marketplace add iris-eval/mcp-server
/plugin install iris-eval-capture@iris-eval
A second, separately installed plugin: three hooks record each turn's prompt, tool calls and final answer and hand them to iris-eval ingest, detached, with critical spans redacted in the stored evaluation text — capture that does not depend on the model deciding to call a tool. It never logs a turn the model already logged, never prints, never blocks, never sends anything anywhere. Installing iris-eval alone changes nothing about your turn loop. Limits and removal: claude-plugin-capture/README.md.
npx -y @iris-eval/mcp-server ingest --file traces.ndjson --evaluate --fail-on detector_vetoingest reads one JSON trace (or NDJSON, one per line) from stdin or a file, stores it, evaluates it under exactly the rules evaluate_output runs, prints one JSON line per trace with the verdict and its basis, and exits 1 when a verdict matches --fail-on. The full recipe, the exit codes and the eight bases are in docs/ci-gate.md.
npx @iris-eval/mcp-server --self-test # offline diagnostic; exit 0 = healthy, 1 = a check failed
npx @iris-eval/mcp-server --version # prints the bare version, e.g. 0.5.1--self-test first creates your Iris home if it is missing and checks that it is writable (exit 1, naming the path, if it is not), then runs its checks — storage round-trip, a planted SSN and a planted injection caught by the safety rules, dashboard boot, the DNS-rebinding guard — inside an isolated temp home, so your real database is never opened. Everything Iris writes lives under one directory, your Iris home: ~/.iris by default (%USERPROFILE%\.iris on Windows), or wherever IRIS_HOME points. That is where iris.db, config.json, custom-rules.json, audit.log, preferences.json and the demo files live; point IRIS_HOME at a scratch directory to try Iris without touching your real data.
Setup by tool
Edit your MCP config file:
- macOS:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%\Claude\claude_desktop_config.json
Add the JSON config above, then restart Claude Desktop.
claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-serverThen restart the session (/clear or relaunch) for tools to load.
Windows note: Do not use
cmd /cwrapper — it causes path parsing issues. Thenpxcommand works directly.
Add to your workspace .cursor/mcp.json or global MCP settings using the JSON config above.
Add to .vscode/mcp.json in your workspace (note: VS Code uses servers, not mcpServers):
{
"servers": {
"iris-eval": {
"command": "npx",
"args": ["@iris-eval/mcp-server"]
}
}
}Open Cline's MCP Servers panel → Configure MCP Servers, and add the mcpServers JSON config above to cline_mcp_settings.json.
Add to Zed settings.json:
{
"context_servers": {
"iris-eval": {
"command": {
"path": "npx",
"args": ["@iris-eval/mcp-server"]
}
}
}
}Add to ~/.codex/config.toml:
[mcp_servers.iris-eval]
command = "npx"
args = ["@iris-eval/mcp-server"]Add the mcpServers JSON config above to ~/.gemini/settings.json.
Iris is a standard stdio MCP server — one npx @iris-eval/mcp-server command, no SDK, no code changes. If your client supports MCP, it supports Iris. Client config formats change; when in doubt, check your client's MCP docs and point it at that command.
# Global install (recommended for persistent data and faster startup)
npm install -g @iris-eval/mcp-server
iris-eval --dashboard
# Docker — two servers, two ports: 3000 = MCP HTTP transport,
# 6920 = dashboard (which also serves the POST /api/v1/traces ingest endpoint).
# The image binds 0.0.0.0 inside the container, so a key is required (see Production).
docker run -p 3000:3000 -p 6920:6920 -v iris-data:/data \
-e IRIS_API_KEY="$(openssl rand -hex 32)" ghcr.io/iris-eval/mcp-serverTip: Global install (
npm install -g) stores traces persistently at~/.iris/iris.db. Withnpx, traces persist in the same location, but startup is slower due to package resolution.
| Trace Logging | Hierarchical span trees with per-tool-call latency, token usage, and cost in USD. Stored in SQLite, queryable instantly. |
| Output Evaluation | 20 built-in rules across 4 categories: completeness, relevance, safety, cost. PII detection (19 patterns: SSN, credit card, phone, email, IBAN, DOB, MRN, IP, API key, passport, plus AWS/Slack/SendGrid/GitHub/Google/npm/DigitalOcean tokens, PEM private-key blocks and seed phrases), prompt injection (37 patterns, phrase + structural), stub-output detection, hallucination detection (25 context-grounded fabrication/contradiction signals — pass input to ground them against the agent's source material), and six trajectory rules that read what the agent DID: an unacknowledged failed tool call, a repeated one (by call, by repeated sequence, or by target once you send tools), a call whose arguments the tool's own JSON Schema rejects and the agent never retried, a file, directory or URL the answer cites that appears in nothing the agent read, an instruction that arrived inside a TOOL RESULT and was then obeyed by a later call, and a task that took more tool calls than your step budget. A trajectory can arrive as tool_calls or as OpenTelemetry TOOL spans. Add custom rules with Zod schemas. |
| LLM-as-Judge | Optional semantic scoring via Anthropic or OpenAI — bring your own API key. Five templates. Hard per-eval cost cap (IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL, default $0.25), per-eval pricing disclosed in the result. |
| Cost Visibility | Aggregate cost across all agents over any time window. Set budget thresholds. Get flagged when agents overspend. |
| Web Dashboard | Real-time dark-mode UI that lands on the failures, worst and newest first — trace visualization, eval results, cost breakdowns, and a command palette (⌘K) that searches your own rules, traces, and evals. |
| Local-first | Everything lives in SQLite on your disk. No account, no sign-up, no telemetry. Outbound HTTP happens only where you opt in: your own LLM-judge key, citation fetching, or an OTel exporter you configure. |
Where this is going next: the roadmap.
Every built-in rule has a published precision, recall and F1 with 95% confidence intervals, measured on a labelled corpus that lives in this repository (proof/corpus/) and regenerates with one command — npm run proof — offline, with no key and no model in the loop. CI re-runs the measurement on every pull request and fails if the committed numbers differ from what the code produces, so a rule cannot change without its numbers changing with it. The numbers are on iris-eval.com/proof and in proof/RESULTS.md; how the corpus was made, what it is not, and how to read an interval are in docs/proof.md. The corpus is synthetic and model-labelled — a human blind label is pending, and the page says so; node proof/blind-sample.mjs draws the reproducible sample that will settle it.
Iris registers twelve tools that any MCP-compatible agent can invoke — trace and rule lifecycle, comparison across runs, LLM-as-judge and semantic citation verification:
log_trace— Log an agent execution with spans, tool calls, token usage, and cost; passevaluate: trueto score it in the same callevaluate_output— Score output quality against completeness, relevance, safety, and cost rules (heuristic, deterministic, free)get_traces— Query stored traces with filtering, pagination, and time-range supportlist_rules— Enumerate deployed custom eval rules (read-only)deploy_rule— Register a new custom eval rule so it fires on everyevaluate_outputof that categorydelete_rule— Remove a deployed custom rule (destructive, idempotent)delete_trace— Remove a single stored trace by ID (destructive, tenant-scoped)evaluate_with_llm_judge— Semantic eval via LLM (Anthropic or OpenAI). Five templates: accuracy, helpfulness, safety, correctness, faithfulness. Cost-capped, per-eval pricing disclosed. Bring your own API key (IRIS_ANTHROPIC_API_KEYorIRIS_OPENAI_API_KEY) — Iris doesn't proxy or relay LLM calls.verify_citations— Extract citations from output (numbered, author-year, URLs, DOIs), fetch sources behind an SSRF-guarded + domain-allowlisted resolver, and use an LLM judge to check whether each source actually supports the cited claim. Opt-in outbound HTTP. Same BYOK requirement asevaluate_with_llm_judge.compare_runs— Did a change make the agent worse? Compares two runs of stored evaluations: a paired exact test when the runs share case keys, an interval on the difference otherwise, and an honest "cannot tell" with the number of cases it would takecompare_traces— How reliably does the agent answer the same question? Per-case pass rates with intervals, flaky cases first, and an overall rate that respects repeatsevaluate_runs— Re-score every trace in a run under today's rules into a new run, so a rules change is never read as an agent change
Enable the LLM judge (optional; the deterministic rules never need it)
- Get an API key from Anthropic or OpenAI.
- Put it in the environment of the process that runs Iris, not only your shell. Claude Code, Claude Desktop, Cursor and most MCP clients: the "env" block of the iris-eval entry in your MCP config — "iris-eval": { "command": "npx", "args": ["-y", "@iris-eval/mcp-server"], "env": { "IRIS_ANTHROPIC_API_KEY": "sk-ant-..." } } (IRIS_OPENAI_API_KEY for an OpenAI key). Docker: -e IRIS_ANTHROPIC_API_KEY=... on the run command. HTTP or CI: export it before starting iris-eval.
- Restart the MCP session. A running process never sees a variable set after it started.
- Confirm from inside your client: read iris://capabilities — judge.enabled must be true there. A key exported in your shell is not passed to the process your client spawns unless its config lists it. On a machine,
npx @iris-eval/mcp-server --self-testprints the judge line for that shell, and GET /api/v1/health reports judge.enabled on a running dashboard. - Spend guard: each call is capped by IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL (default 0.25 USD) and refused before any spend if the worst case would exceed it. Iris calls the provider directly with your key and never proxies it.
When IRIS_OTEL_ENDPOINT is configured, log_trace calls also emit a best-effort OTLP/HTTP JSON export to any OpenTelemetry collector (Jaeger, Grafana Tempo, Datadog OTLP, Honeycomb, etc). See docs/otel-integration.md.
evaluate_output returns both a score and a passed flag — they answer different questions:
score(0..1) is the weighted average across the rules that ran — a quality gradient.passedis the ship/no-ship verdict, and the score is never consulted for it. A composer reads each rule by the kind of claim it makes: a policy you configured gates; a critical detector vetoes; a critical check that was asked and could not answer makes the verdict unknown (passed: false) rather than clean; every remaining detector combines into one probability that the output is bad, weighed against the loss ratio you state ineval.falsePassCost(default 1, so the cut is 0.5).verdict.basisnames the layer that decided andverdict.bythe rules;interpretations[]says why a rule that failed did not decide and which setting would change that, and names any question that was not judged and the input that would let it be.
Genuine safety violations hard-fail. By default no_pii, no_injection_patterns, and no_blocklist_words are critical rules: if one fails, the eval reports passed: false no matter how well the other rules scored, and the response names the culprits in critical_failures. A leaked SSN can't be averaged away. Which built-in rules are critical is a deployment setting (eval.criticalRules / eval.nonCriticalRules); every rule result carries the effective critical flag and criticalSource, and list_rules reports the roster this server applies. Custom rules deployed with severity: "high" or "critical" hard-fail the same way; low/medium severities only affect the score. One boundary to know, stated the same way on every surface: a critical rule that skipped (missing context, a broken definition, or a regex killed at the sandbox budget) has not judged the output and does not veto — every such rule is named in critical_skipped. A gate that must fail closed treats a non-empty critical_skipped as unknown, not clean, and may treat any budgetExceeded skip in rule_results the same way.
For CI gates: if you omit eval_type, every bundle runs — completeness, relevance, safety, cost and any custom rules — and the response says eval_type: "all" with a note that the default ran, plus a per-bundle categories map. A bundle with nothing to judge (cost without cost_usd, relevance without input) reports passed: null there — not evaluated, not failing — and never counts toward the verdict. The response always echoes the eval_type that ran, so your gate can verify coverage; key on passed for the verdict and name a bundle only when you want a narrower run.
Two ways to add a rule. Inline rules ride along on one evaluate_output call (custom_rules, up to 10 per call); they fire alongside whatever eval_type bundle you chose, or alone with eval_type: "custom". Deployed rules are registered once with deploy_rule, persist in custom-rules.json under your Iris home, and fire on every future evaluate_output of their evalType. The definition is the same shape either way:
| Field | Required | What it is |
|---|---|---|
name |
yes | 1–80 characters; appears as ruleName in results |
type |
yes | one of regex_match · regex_no_match · min_length · max_length · contains_keywords · excludes_keywords · json_schema · cost_threshold |
config |
yes | the keys for that type: pattern (+ optional flags) for the two regex types · min_length / max_length (a character count) · keywords (+ optional threshold, 0–1, default 1 = all must appear) for the two keyword types · {} for json_schema · max_cost in USD for cost_threshold |
weight |
no | weight in the score; default 1 |
deploy_rule wraps the definition with name, an optional description, evalType (completeness · relevance · safety · cost · custom) and severity. Severity says what a failure means: low/medium only lower the score; high/critical hard-fail the evaluation — passed: false, the rule named in critical_failures — whatever the weighted score says. A rule that skips (a cost_threshold rule with no cost_usd, or a regex killed at the 100 ms sandbox budget) has not judged the output and is listed in critical_skipped instead. Deploy a critical rule that forbids internal hostnames in anything the agent says:
{
"name": "no_internal_hostnames",
"description": "Output must not mention internal hostnames.",
"evalType": "safety",
"severity": "critical",
"definition": {
"name": "no_internal_hostnames",
"type": "regex_no_match",
"config": { "pattern": "\\b[a-z0-9-]+\\.internal\\.example\\b", "flags": "i" }
}
}The response is the persisted rule — keep the id for delete_rule:
{ "rule": { "id": "rule-588823d0", "name": "no_internal_hostnames", "evalType": "safety", "severity": "critical", "enabled": true, "version": 1, "definition": { "…": "…" } } }From the very next evaluate_output with eval_type: "safety", an output that mentions db-primary.internal.example comes back passed: false with critical_failures: ["no_internal_hostnames"] — even though all five built-in safety rules passed and the weighted score is 0.895. Regex patterns must pass a ReDoS check at deploy time and always run in a sandbox worker under a hard 100 ms deadline. list_rules shows what is deployed; the dashboard's rule composer builds the same shape from a failure you clicked on. Full reference, scoring per type, and worked examples: docs/custom-rules.md.
Full tool schemas and configuration: iris-eval.com
Iris runs entirely on your machine today, and everything it does is free and MIT licensed with no limits and no account.
Hosted storage, shared team history and alerting are under consideration, not under construction. There is no pricing, and nothing to buy. If shared history would be useful to you, the waitlist is how we find out whether it's worth building — it commits you to nothing.
Two commitments hold regardless: nothing that is free today will move behind a paywall, and no compliance certification will be claimed before it is held.
- Claude Desktop setup — MCP config for stdio and HTTP modes
- TypeScript — MCP SDK client — connect and invoke tools
- HTTP transport (TS + Python) — full client code for REST-style integration
- LangChain instrumentation (Python, conceptual) — scaffold showing the shape; needs your agent code to be runnable
- CrewAI instrumentation (Python, conceptual) — scaffold; same caveat
- GitHub Issues — Bug reports and feature requests
- GitHub Discussions — Questions and ideas
- Contributing Guide — How to contribute
- HTTP Ingest — Deterministic trace capture via
POST /api/v1/traces - Roadmap — What's coming next
- Versioning policy — What each version number promises, and what has to be true before 1.0
Configuration & Security
| Flag | Default | Description |
|---|---|---|
--transport |
stdio |
Transport type: stdio or http |
--port |
3000 |
HTTP transport port |
--db-path |
~/.iris/iris.db |
SQLite database path |
--config |
~/.iris/config.json |
Config file path |
--api-key |
— | API key for HTTP authentication (transport and dashboard, including POST /api/v1/traces) |
--dashboard |
false |
Enable web dashboard. Also the only way the POST /api/v1/traces ingest endpoint starts — it never starts implicitly with --transport http |
--dashboard-port |
6920 |
Dashboard port |
--dashboard-host |
127.0.0.1 |
Dashboard bind address. Loopback by default — the dashboard is unauthenticated unless --api-key is set, so binding beyond loopback exposes your full trace history |
--demo |
false |
Seed a demo database (separate from your real traces) and serve the dashboard against it |
--demo-clear |
false |
Delete the demo database and exit |
--self-test |
false |
Run the offline install diagnostic in an isolated temp home, then exit (0 = healthy, 1 = a check failed) |
--purge |
false |
Delete every stored trace, span and evaluation from the configured database, compact the file and truncate the write-ahead log so the deleted text does not linger on disk, then exit. Deployed rules, the audit log and preferences are kept. Not reversible. Stop any running Iris server first — the file is compacted in place. Refuses to combine with --demo, --demo-clear or --self-test |
--version |
— | Print the bare version (e.g. 0.5.1) to stdout and exit 0. Reads nothing under your Iris home |
Every variable --help documents. CLI flags take precedence over environment variables when both are set.
| Variable | Description |
|---|---|
IRIS_TRANSPORT |
Transport type (stdio or http) |
IRIS_HOST |
HTTP transport bind address (default 127.0.0.1) |
IRIS_PORT |
HTTP transport port (1-65535, default 3000) |
IRIS_HOME |
Directory for all per-user files: config.json, iris.db, custom-rules.json, audit.log, preferences.json (default ~/.iris) |
IRIS_DB_PATH |
SQLite database path (overrides IRIS_HOME for the DB only) |
IRIS_LOG_LEVEL |
Log level: debug, info, warn, error |
IRIS_DASHBOARD |
true/1/yes/on enables the web dashboard; false/0/no/off disables it (also overrides dashboard.enabled in config.json) |
IRIS_DASHBOARD_PORT |
Dashboard port (1-65535, default 6920) |
IRIS_DASHBOARD_HOST |
Dashboard bind address (default 127.0.0.1) |
IRIS_API_KEY |
API key for HTTP authentication. Required to bind the HTTP transport or the dashboard beyond loopback (0.0.0.0, a LAN address, a container): without it the server refuses to start |
IRIS_ALLOW_UNAUTHENTICATED |
Set to 1 to run a non-loopback bind with no key on purpose (lifts the refusal; the network is then your boundary) |
IRIS_ALLOWED_ORIGINS |
Comma-separated origin allowlist. Dashboard: CORS headers (supports globs, e.g. http://localhost:*). HTTP transport: exact-match Origin allowlist for DNS-rebinding protection (globs ignored; the server's own loopback origins are always allowed) |
IRIS_NO_AUTO_LAUNCH |
Set to 1 to disable the first-run dashboard auto-launch |
IRIS_ANTHROPIC_API_KEY |
Required by evaluate_with_llm_judge + verify_citations with provider=anthropic |
IRIS_OPENAI_API_KEY |
Required by evaluate_with_llm_judge + verify_citations with provider=openai |
IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL |
Hard cost cap per LLM judge call (default 0.25) |
IRIS_CITATION_ALLOW_FETCH |
Set to 1 to permit outbound HTTP in verify_citations (off by default) |
IRIS_CITATION_DOMAINS |
Comma-separated hostname allowlist for verify_citations (suffix match) |
IRIS_OTEL_ENDPOINT |
Enable best-effort OTLP/HTTP JSON trace export to this collector URL |
IRIS_OTEL_SERVICE_NAME |
service.name resource attribute for OTel export (default iris-eval) |
IRIS_OTEL_HEADERS |
Comma-separated k=v headers for OTel export (e.g. authorization=Bearer abc) |
IRIS_OTEL_TIMEOUT_MS |
Per-export timeout (default 15000) |
RATE_LIMIT_SALT |
Website waitlist API only — required when the iris-eval.com site is deployed; the server never reads it |
When using HTTP transport, Iris includes:
- API key authentication with timing-safe comparison (Bearer for API clients; browser sign-in to the dashboard via
?key=) - CORS restricted to localhost by default
- Rate limiting (600 req/min dashboard API, 20 req/min MCP)
- Helmet security headers
- Zod input validation on all routes
- ReDoS-safe regex for custom eval rules
- 1MB request body limits
# Production deployment
iris-eval --transport http --port 3000 --api-key "$(openssl rand -hex 32)" --dashboardWith a key set, API clients — MCP clients, capture SDKs, POST /api/v1/traces — send Authorization: Bearer <key>. To open the dashboard in a browser, append the key once to any dashboard URL, http://localhost:6920/?key=<api key>: Iris exchanges it for an HttpOnly, SameSite=Lax session cookie and redirects to the same page with the key removed from the address bar. A page opened without a session shows a sign-in form that does the same exchange. The key is never stored in the browser, and sessions live only in the server process (at most 256 live at a time; a sign-in that finds them all live is refused rather than evicting one).
Iris refuses to start when the HTTP transport or the dashboard is bound beyond loopback — 0.0.0.0, a LAN address, a container — with no API key, and says so in one sentence naming IRIS_API_KEY. That includes a bare docker run of the image, which binds 0.0.0.0 inside the container because loopback is unreachable through a published port. Loopback without a key keeps working (with a warning on the HTTP transport): the machine boundary is the exposure control there.
# The image: pass a key
docker run -p 3000:3000 -p 6920:6920 -v iris-data:/data \
-e IRIS_API_KEY="$(openssl rand -hex 32)" ghcr.io/iris-eval/mcp-server
# Compose: the file requires the variable and refuses before the container starts
IRIS_API_KEY="$(openssl rand -hex 32)" docker compose up
# A network you have already fenced some other way: run open, on purpose
IRIS_ALLOW_UNAUTHENTICATED=1 iris-eval --transport http --dashboardOpen by design, on a keyed server: GET /health on the transport and GET /api/v1/health on the dashboard answer without a key — status, version, uptime, storage connectivity, whether a judge key is present — never the key, never a trace. Everything else needs Authorization: Bearer <key> or a browser session. Retention runs on every server: traces and evaluations older than retention.days (default 30) are deleted at startup and every retention.sweepIntervalHours; --self-test prints this install's policy, and iris://capabilities / GET /api/v1/capabilities carry it as retention.
Everything Iris stores lives under your Iris home (~/.iris, or IRIS_HOME). iris.db keeps every trace's input and output verbatim — including any text no_pii goes on to flag; detection does not redact unless you ask it to: storage.redact: "critical_spans" in config.json stores each evaluation's output with the spans a critical detector flagged replaced by [REDACTED:<pattern>] (off by default; the evidence offsets still index the text the caller saw). At startup, and every retention.sweepIntervalHours (default 24, 0 disables the timer) after that, traces and evaluations older than retention.days (default 30, 0 disables, set in config.json) are deleted and the write-ahead log is checkpointed. Deleting a trace — by delete_trace or by the sweep — erases the text of every evaluation linked to it (the output, the expected text, the suggestions and the rule messages) and stamps erased_at; the verdict, the scores and the evidence offsets stay. To remove everything now, stop the server and run --purge: it deletes every stored trace, span and evaluation, compacts the database and truncates the write-ahead log so the text is gone from disk, and keeps your deployed rules, audit log and preferences.
Troubleshooting
npx @iris-eval/mcp-server --self-testIt checks storage, the deterministic evals, and the dashboard in an isolated temp home and prints a per-step verdict — the failure output names the broken step. Exit code 0 means the install is healthy.
You may have a cached older version. Clear the npx cache and retry:
npx --yes @iris-eval/mcp-server@latestOr install globally to avoid cache issues entirely:
npm install -g @iris-eval/mcp-server@latestIris stores traces with better-sqlite3, a native module that fetches or compiles its binding in an install script. If that script was skipped — --ignore-scripts on the command line, ignore-scripts=true in an .npmrc (common on corporate machines), or a registry mirror that strips postinstall — startup fails with a long "Could not locate the bindings file" dump listing a dozen paths it tried. Rebuild that one module:
npm rebuild better-sqlite3
# for a global install:
npm rebuild -g better-sqlite3MCP tools only load at session start. After adding iris-eval, restart the session with /clear or relaunch the terminal.
npx @iris-eval/mcp-server --versionThe first startup log line also carries it (Starting Iris MCP server vX.Y.Z), and --self-test prints it in its summary. For a global install, npm ls -g @iris-eval/mcp-server shows the installed version.
# If using npx (clears cache and fetches latest)
npx --yes @iris-eval/mcp-server@latest
# If installed globally
npm update -g @iris-eval/mcp-serverIris requires Node.js 20 or later. Node 18 reached EOL in April 2025 and is not supported.
node --version # Must be v20.x or v22.x+Claude Code's /doctor may suggest wrapping npx with cmd /c. This is not needed and causes path parsing issues. Use npx directly:
# Correct
claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-server
# Wrong (causes /c to be parsed as a path)
claude mcp add --transport stdio iris-eval -- cmd /c "npx @iris-eval/mcp-server"If Iris is useful to you, consider starring the repo — it helps others find it.
MIT Licensed.