Acceptance testing of the packed tarball from a clean install (no repo checkout) surfaced a cluster of README/docs truth gaps:
- Every relative link is dead from the install.
docs/http-ingest.md, docs/roadmap.md, docs/otel-integration.md, examples/*, CONTRIBUTING.md — none ship in the npm package (only dist/, LICENSE, README.md, server.json). The ingest doc is the designated non-MCP integration path and is unreachable from the artifact that recommends it. Fix: absolute GitHub URLs (like the badges already use), or add docs/ to the files array.
- The README never says
POST /api/v1/traces lives on the DASHBOARD port (default 6920), not the transport port. First attempt 404s for anyone reading --transport http.
- Env-var table lists 9 vars;
--help documents ~25. Missing rows include the ones the README itself references (IRIS_ANTHROPIC_API_KEY, IRIS_OPENAI_API_KEY, IRIS_OTEL_ENDPOINT) plus IRIS_HOME, IRIS_NO_AUTO_LAUNCH, IRIS_DASHBOARD_HOST, and the two IRIS_CITATION_* egress switches. Add a drift test comparing the table to the --help block (model on the claims-drift tests).
- Security section claims "100 req/min API"; the server enforces and advertises 600 (
src/config/defaults.ts). Verified: 120 POSTs in ~15s all 201. Also mirror in docs/architecture.md:474.
- Troubleshooting "Version check" recipe is false:
--help prints no version and there is no --version flag. Interpolate PKG_VERSION into the help banner + add --version, or change the recipe to --self-test.
--self-test and IRIS_HOME — the two features testers praised most — are undocumented in the README. Add a "Verify your install" line.- Corporate
npm install --ignore-scripts breaks the native better-sqlite3 binding and the failure is a 13-path bindings dump. One Troubleshooting line (npm rebuild better-sqlite3) plus a friendlier catch in storage init.
- The "Install in Cursor" badge registers the server under the name
server (deeplink name=server) instead of iris-eval. README line 4.
- No "Authoring a custom rule" section: the definition shape (
definition.type values, config keys, weight) is documented only inside the MCP tool description, invisible unless your client renders it.
Found during install-only acceptance testing of the v0.5.0 release candidate (packed tarball, no repo checkout). Part of a 9-issue batch; the blocker + core MAJOR cluster shipped in #365–#368 incl. the sandbox in #366.
Acceptance testing of the packed tarball from a clean install (no repo checkout) surfaced a cluster of README/docs truth gaps:
docs/http-ingest.md,docs/roadmap.md,docs/otel-integration.md,examples/*,CONTRIBUTING.md— none ship in the npm package (onlydist/,LICENSE,README.md,server.json). The ingest doc is the designated non-MCP integration path and is unreachable from the artifact that recommends it. Fix: absolute GitHub URLs (like the badges already use), or adddocs/to thefilesarray.POST /api/v1/traceslives on the DASHBOARD port (default 6920), not the transport port. First attempt 404s for anyone reading--transport http.--helpdocuments ~25. Missing rows include the ones the README itself references (IRIS_ANTHROPIC_API_KEY,IRIS_OPENAI_API_KEY,IRIS_OTEL_ENDPOINT) plusIRIS_HOME,IRIS_NO_AUTO_LAUNCH,IRIS_DASHBOARD_HOST, and the twoIRIS_CITATION_*egress switches. Add a drift test comparing the table to the--helpblock (model on the claims-drift tests).src/config/defaults.ts). Verified: 120 POSTs in ~15s all 201. Also mirror indocs/architecture.md:474.--helpprints no version and there is no--versionflag. InterpolatePKG_VERSIONinto the help banner + add--version, or change the recipe to--self-test.--self-testandIRIS_HOME— the two features testers praised most — are undocumented in the README. Add a "Verify your install" line.npm install --ignore-scriptsbreaks the native better-sqlite3 binding and the failure is a 13-path bindings dump. One Troubleshooting line (npm rebuild better-sqlite3) plus a friendlier catch in storage init.server(deeplinkname=server) instead ofiris-eval. README line 4.definition.typevalues, config keys, weight) is documented only inside the MCP tool description, invisible unless your client renders it.Found during install-only acceptance testing of the v0.5.0 release candidate (packed tarball, no repo checkout). Part of a 9-issue batch; the blocker + core MAJOR cluster shipped in #365–#368 incl. the sandbox in #366.