test(examples): OpenInference active-ingestion conformance fixture (TH-8336) by nik13 · Pull Request #227 · future-agi/traceAI

nik13 · 2026-10-05T16:28:13Z

Summary

This PR adds an OpenInference active-ingestion conformance fixture at python/examples/openinference/ (Linear TH-8336, decision D-8336). There is no package, no Go port, and no edit to the backend's openinference.py or otel_compat_urls.py.

Fixture: tests/fixtures/openinference-trace.otlp.json, one OTLP/JSON request with three spans in one trace:
- retrieve: openinference.span.kind=RETRIEVER, with the query in input.value;
- ChatCompletion: LLM, llm.model_name, llm.token_count.prompt/completion;
- lookup_refund_policy: TOOL.
The resource carries project_name and project_type=observe.
Golden: tests/fixtures/openinference-spans.golden.json, the posted spans verbatim.
Tests: tests/test_openinference_fixture.py. It posts the fixture with the shared harness post_otlp() and checks it with compare() against the golden. Four opt-in tests (FI_COLLECTOR_SRC) match the fi-collector source statements behind each source-reading fact, with comments stripped. They match source text and do not run the collector.
README: the Django route is retired and is not proof (futureagi/tracer/otel_compat_urls.py:15-16 holds the route only as a comment). It separates what is tested from what is read in source.

What fi-collector does with these keys (source reading at future-agi `main` 4af5338)

Fixture key	fi-collector	Stored as
`openinference.span.kind`	4th kind key (`exporter/clickhouse25exporter/converter.go:79-84`), lower-cased and validated (`:107-131`)	`observation_type` `retriever` / `llm` / `tool`
`llm.model_name`	first model alias (`pkg/adapter/adapter.go:279-284`)	`model`
`llm.token_count.prompt/completion`	first token aliases (`adapter.go:296-305`), total summed (`:230-235`)	`prompt_tokens`, `completion_tokens`
`input.value`	overflow (`adapter.go:32-40`) and lifted into `input` (`converter.go:343`)	`attributes_extra` and `input`

On an authenticated request, a batch whose resource has no project_name fails with HTTP 400 (pkg/auth/stamp.go:31-46, server.go:458-465); a collector with auth disabled skips that check. A project_name that is not yet a project in the key's workspace is created (pkg/auth/auth.go:202-216). One that still does not resolve is not rejected: its spans are dropped and the request is answered 200, with only a collector-log warning (stamp.go:57-85, server.go:466-468).

Unknown kinds (PROMPT, DECISION, UNKNOWN, empty) are stored as unknown. The function that does this has no error path, so a kind value cannot fail an export. How the trace view renders observation_type is shared processor work (SF-1) and is not claimed here.

Deviations from the spec (approved: Nikhil 2026-10-03 blanket)

D1, PRD premise false: the PRD says fi-collector does not map openinference.span.kind. It does, via the converter (not DeriveHotKeys), and did already at d794a49. fi-collector's own test pins "Retriever" → "retriever" (converter_test.go:784). Nothing to fix; only the display remains SF-1.
D2: input.value is overflow and lifted into the input column, so no new column is needed.
D3, license: the legacy License field of openinference-instrumentation 0.1.70 is null, but the wheel declares License-Expression: Apache-2.0. The README claims no license, as the architecture says.
D4, AC-01 round trip (post to a live fi-collector and read back): not done; the architecture limits this card to the harness. As a partial substitute outside the repo tests, the builder ran fi-collector 4af5338's own decoder and ConvertWithIdentities offline on the fixture and on unknown-kind variants. That run used no server, auth or ClickHouse.
D5: "an unknown kind must not 4xx" is weak against the harness Receiver, which never reads kinds. The collector side rests on source plus that offline run, and the README says so.
IDs must be hex: fi-collector decodes OTLP/JSON with pdata UnmarshalJSON. Base64 IDs (what protobuf MessageToDict produces) fail with 400 (server.go:438-442). A test pins hex IDs.

Review

Round 1: pr-reviewer (APPROVE) and pr-verifier (VERIFIED) at 7fd83eb, with no P1/P2 findings. The P3s that made the README claim more than the tests prove are fixed in ba6cc08..ea8859b:
- R1: the project_name 400 is qualified (authenticated path only), and the drop-with-200 case is stated.
- V1: the opt-in tests now match the fi-collector statements behind each fact (Split, DeriveHotKeys, resolveObservationType, spanToRow, handleHTTPTraces, StampResourceAttrs, ResolveProjectsForKey), with comments stripped. In the builder's mutation run they catch 27 of 27 source mutations; the 7fd83eb tests caught 10. The README now says they match source text and do not run the collector.
- R5: an ast scan of every .py file in the example and the harness rejects openinference/tracer imports, including ones that never run. The run-time check stays.
Round 2: pr-verifier t_353b2a8e VERIFIED at ea8859b, no P1/P2. Retained P3s (narrated in the video, not fixed here): V2 a failed project auto-create drops the batch with HTTP 200, while lookup/scope failures return 400; V3 project_type is an end-user gate, and its absence is not an error; V4 the import scan does not catch importorskip or dynamic/non-literal imports.
Follow-ups outside this PR: the stale spec/docs text (kind already mapped, license) is corrected in company-brain Dev #63 (886fe7d, open); a converter_test.go case that decodes this fixture is future-agi work.

Tests

Head ea8859b. Coordinator rerun from the repo root on a clean tree (tickets/TH-8336/exact-head-ea8859b.txt):

env -u PYTHONPATH PYTHONPATH="python:python/tests" uv run --no-project --python <py> \
  --with 'pytest==9.1.1' --with 'opentelemetry-sdk==1.45.0' --with 'opentelemetry-exporter-otlp-proto-http==1.45.0' \
  --with 'opentelemetry-instrumentation==0.66b0' --with 'requests==2.34.2' --with 'jsonschema==4.26.0' \
  pytest python/examples/openinference/tests -q -p no:cacheprovider --noconftest -o addopts= -rfEs
              default                FI_COLLECTOR_SRC=<fi-collector @ 4af5338>
Python 3.10   21 passed, 4 skipped   25 passed
Python 3.11   21 passed, 4 skipped   25 passed
Python 3.12   21 passed, 4 skipped   25 passed
Python 3.13   21 passed, 4 skipped   25 passed

4 files, all under python/examples/openinference/. 0 secret-like added lines; 0 files outside the directory.
The tests post only to the harness Receiver on 127.0.0.1. There is no fi-collector, Arize account or live instrumentor.

Coverage:

golden match, with a compare() control (a kind's case change fails it);
tree and parentage;
kind strings as posted, and no other kind key;
model and token keys and types, and the query in input.value;
hex IDs and the resource project_name;
unknown kinds post unchanged;
post_otlp() refuses non-loopback endpoints;
no openinference or tracer import in the example or harness source (ast scan with a control), and none loaded at run time;
no key material in the fixture.

Limits

The Receiver asserts what is posted, not what is stored. Auth (401), project stamping, storage and the trace view need a collector, which the harness does not start.
The PRD's privacy and tenant checks (two projects) are not in the architecture and are not done.
No CI job runs this directory. Run the opt-in tests whenever fi-collector's kind, alias or project code changes. They are text checks: a refactor that keeps behaviour fails them, and a behaviour change outside the pinned statements passes them.

Video demo

Recorded build: head ea8859be850ced2f6b2b6fa80828d07c1986ad01 (this PR); real terminal chapters from a clean read-only worktree; 7:00, 11 chapters, narrated with captions.

The video (SHA-256 8bf39ecd796b5f84c3bd86a18a564a4af23e9e51da73b16096fc91843a63cdfe), captions (SRT), transcript, chapter list, preview, recorded driver output and both media-verification notes are private attachments on Linear issue TH-8336 (Future AGI workspace; sign-in required). They are not linked here as raw upload URLs.

Time	Chapter
00:00	Problem and scope
00:33	Verified build and live remote
01:04	Three-span fixture and parent tree
01:39	Real loopback post and negative control
02:19	Pinned source: kind mapping
03:02	Pinned source: model, tokens and input
03:37	Pinned source: qualified project outcomes
04:21	Four opt-in tests and real mutation
04:58	AST import guard and disclosed gaps
05:39	README test commands and clean tree
06:15	Review verdict and remaining limits

Independent media checks (Rick): full audio/video decode exit 0; 11 embedded chapters match the list; the only audio gaps (about 1.5 s) are the 10 chapter transitions; every end-of-chapter frame inspected (loopback port and temp paths shown as placeholders, header names only); 8 delivered files scanned with zero secret/path/port/header hits. Disclosed edits (recorder): two explanatory cards, reading holds, cropped/panned source views, and one chapter re-recorded at a smaller font so its completion marker fits. The video shows the harness Receiver and pinned collector source reading; it does not show a running fi-collector, storage, auth, tenants or the trace view.

Stacked on #203 (shared harness, 3eaadc8), base dev. Not merged.

Linear: TH-8336. approved: Nikhil 2026-10-03 blanket

The Node OTLP exporter streams with Transfer-Encoding: chunked and sends no Content-Length. The Receiver read Content-Length only, so every Node export came back 400 with zero spans; two TH-8103 children had to put a de-chunking relay in front of it. Receiver now accepts both framings. It also records one entry per accepted export (path, lower-cased headers, flattened resource attributes) via requests(), so a contract test can assert X-Api-Key/X-Secret-Key and project_name through the shared harness instead of a private recorder. Verified: 6 harness tests pass, and the TanStack example's real Node exporter delivered 4 chunked exports (4 spans, collector path, both auth headers, project_name/project_type) with no relay. Refs: TH-8339, TH-8103

Three spans in one trace, OTLP/JSON as fi-collector's HTTP handler decodes it (hex trace and span ids, int64 values as decimal strings): - retrieve: openinference.span.kind=RETRIEVER, input.value (the root) - ChatCompletion: openinference.span.kind=LLM, llm.model_name, llm.token_count.prompt, llm.token_count.completion (child of retrieve) - lookup_refund_policy: openinference.span.kind=TOOL (child of retrieve) The resource carries project_name, which fi-collector requires (pkg/auth/stamp.go:31-45), and project_type=observe. The keys are the ones openinference-semantic-conventions 0.1.41 defines and adapter.go / converter.go name; nothing imports openinference-instrumentation. openinference-spans.golden.json is the posted body's spans, verbatim, not a rendered trace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8336

States that the Django OTLP route is retired and is not proof (future-agi main 4af5338 otel_compat_urls.py:15-16: the only route is commented out), and what the fixture posts and why it is encoded as it is: hex ids, because fi-collector's pdata JSON decoder rejects base64 (server.go:438-442). Records, as source reading of fi-collector 4af5338, that the collector already reads every key the fixture sets: openinference.span.kind is the fourth kind key (converter.go:79-84, lower-cased and validated at :107-131), llm.model_name and llm.token_count.prompt/completion are the first aliases (adapter.go:279-305), and input.value is overflow (adapter.go:32-40) lifted into the input column (converter.go:343). Kind values outside the type list, including OpenInference's PROMPT and DECISION, are stored as unknown with no error path (converter.go:127-129, :407). Display is SF-1 and not claimed. Pins openinference-instrumentation 0.1.70 for the later round trip only: not installed, not imported, no license claimed. Tested Pythons are 3.11 and 3.13; 3.10 and 3.12 are stated as untested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8336

Posts fixtures/openinference-trace.otlp.json with post_otlp() to the shared harness Receiver and checks the decoded spans with compare() against the golden, plus exact equality (ids, parentage, times). Every always-on assertion is about the posted body, since the Receiver does not authenticate, resolve kinds or store anything: - one export to /v1/traces with no key headers; the resource carries project_name and project_type=observe; - one trace, retrieve the parent of ChatCompletion and lookup_refund_policy; ids are hex as fi-collector's JSON decoder needs; - kind strings posted verbatim (RETRIEVER, LLM, TOOL) and no other key the collector reads a kind from; a control shows compare() fails when a kind value's case changes; - the LLM span's model and intValue token keys, the retriever's query in input.value, the tool span's kind only; - kinds PROMPT, DECISION, "" and not-a-kind post without error and arrive unchanged (the docstring says the Receiver never reads kinds); - post_otlp() refuses non-loopback endpoints before urlopen, with a check that the patched urlopen is on the real path; no openinference or tracer module is imported; the fixture holds no key material. Two opt-in tests (FI_COLLECTOR_SRC) read fi-collector's Go tables (spanKindAttrKeys, knownObservationTypes, spanKindSynonyms, the model and token alias lists, overflowKeyPrefixes) rather than copying them. They pass against 4af5338 and fail against copies with the kind key removed or "prompt" added to the type list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8336

The four evidence runs on Python 3.11 and 3.13 took 2.7-3.0 s each, not about a second. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8336

Coordinator exact-head run at 5be86d4 passed on 3.10, 3.11, 3.12 and 3.13 (18 passed + 2 skipped default; 20 passed with FI_COLLECTOR_SRC). approved: Nikhil 2026-10-03 blanket Refs: TH-8336

…orts The sys.modules check could not fail with the README command: neither openinference nor the backend's tracer package is installed. Add an ast scan of every .py file in the example and the harness for imports of either (import, from-import, import_module/__import__ of a literal), with a control, and keep the run-time check for environments that have them. The README says what each one guards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8336

…nce facts The opt-in source tests pinned key tables and literals, so edits that falsify README facts passed them (verifier V1: project_name fail-fast disabled, unknown kind returning "", observation_type not from the resolver, total_tokens derivation removed, kind key moved first). Read function bodies with comments removed and whitespace collapsed and match the statements that carry each fact: Split routing, DeriveHotKeys model/token/total, resolveObservationType's first-key, lower-case, synonym and unknown fallback, spanToRow's stored columns and single nil error return, the kind key order, and the project_name statements in handleHTTPTraces, StampResourceAttrs and ResolveProjectsForKey. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8336

…the drop The README said fi-collector fails a batch without project_name with a 400, unqualified. That holds only on an authenticated request: stamping is gated on auth.FromContext (server.go:458, stamp.go:19-21). It also left out that a project_name that is created on demand but still does not resolve is dropped with a 200 and a log warning, not rejected (auth.go:202-216, stamp.go:57-85, server.go:466-468). State both, and guard the wording with a README test; the opt-in test reads the lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8336

The README said the opt-in tests "check that the kind, model, token, input.value and project_name facts above still hold". They match source text, not behaviour. List the tables and statements they read, say how they match, and that a behaviour change outside those statements passes them while a behaviour-preserving refactor fails them. Guard the wording with a README test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8336

nik13 and others added 12 commits October 3, 2026 18:58

test(harness): add shared OTLP harness for TH-8103 contract tests

5936e09

test(harness): accept the collector /tracer/v1/traces path

32f5c9d

docs(examples): state the measured OpenInference test run time

5be86d4

The four evidence runs on Python 3.11 and 3.13 took 2.7-3.0 s each, not about a second. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8336

docs(examples): OpenInference fixture runs on Python 3.10-3.13

7fd83eb

Coordinator exact-head run at 5be86d4 passed on 3.10, 3.11, 3.12 and 3.13 (18 passed + 2 skipped default; 20 passed with FI_COLLECTOR_SRC). approved: Nikhil 2026-10-03 blanket Refs: TH-8336

Navigation Menu

Repository navigation

test(examples): OpenInference active-ingestion conformance fixture (TH-8336) - #227

test(examples): OpenInference active-ingestion conformance fixture (TH-8336)#227
nik13 wants to merge 12 commits into
devfrom
feat/th-8336-openinference

nik13 commented Oct 5, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

nik13 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What fi-collector does with these keys (source reading at future-agi main 4af5338)

Deviations from the spec (approved: Nikhil 2026-10-03 blanket)

Review

Tests

Limits

Video demo

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

nik13 commented Oct 5, 2026 •

edited

Loading

What fi-collector does with these keys (source reading at future-agi `main` 4af5338)