Objective
Remove the input_segments field from EvalTest by resolving file content directly into input (TestMessage[]) at parse time. Today input and input_segments carry overlapping data — input_segments is a flattened, role-stripped version of input with resolved file text. This duplication caused a bug where {{ input }} in judge templates showed file paths without content (fixed in PR #507 via a render-time patch). Consolidating eliminates the duplication and the patch.
Architecture Boundary
core-runtime — touches the EvalTest type contract, YAML/JSONL parsers, prompt-builder, and evaluator variable binding. All changes are within packages/core/src/evaluation/ and apps/cli/src/commands/trace/.
Current State
| Concern |
Where it reads from today |
| Judge template `{{ input }}` |
evalCase.input + render-time resolveInputWithFileContent() patch (llm-judge.ts) |
| Multi-turn prompt building |
evalCase.input for messages, evalCase.input_segments for file content lookup |
| Single-turn prompt building |
evalCase.input_segments directly (prompt-builder.ts:165) |
| Trace scoring file map |
evalCase.input_segments for resolvedPath (trace/score.ts) |
Design Latitude
- Must: enrich
input message content segments with text (file content) and resolvedPath during processMessages() in yaml-parser/jsonl-parser
- Must: update prompt-builder single-turn path to derive segments from
input instead of input_segments
- Must: update trace scoring to extract file paths from
input
- Must: remove
input_segments from EvalTest interface, all parsers, and test fixtures (~18 files)
- Must: remove
resolveInputWithFileContent() from llm-judge.ts (the patch becomes unnecessary)
- Free to decide: how to structure the flattening helper for single-turn prompt-builder (inline loop vs extracted utility)
- Free to decide: whether
resolvedPath lives on the content segment or is derived at point of use
Acceptance Signals
Non-Goals
- Changing the YAML/JSONL user-facing schema (input format stays the same)
- Modifying the
TestMessage type definition (content segments are already JsonObject)
- Changing how guidelines are extracted/filtered (that stays in
processMessages)
Related
Objective
Remove the
input_segmentsfield fromEvalTestby resolving file content directly intoinput(TestMessage[]) at parse time. Todayinputandinput_segmentscarry overlapping data —input_segmentsis a flattened, role-stripped version ofinputwith resolved file text. This duplication caused a bug where{{ input }}in judge templates showed file paths without content (fixed in PR #507 via a render-time patch). Consolidating eliminates the duplication and the patch.Architecture Boundary
core-runtime— touches the EvalTest type contract, YAML/JSONL parsers, prompt-builder, and evaluator variable binding. All changes are withinpackages/core/src/evaluation/andapps/cli/src/commands/trace/.Current State
evalCase.input+ render-timeresolveInputWithFileContent()patch (llm-judge.ts)evalCase.inputfor messages,evalCase.input_segmentsfor file content lookupevalCase.input_segmentsdirectly (prompt-builder.ts:165)evalCase.input_segmentsforresolvedPath(trace/score.ts)Design Latitude
inputmessage content segments withtext(file content) andresolvedPathduringprocessMessages()in yaml-parser/jsonl-parserinputinstead ofinput_segmentsinputinput_segmentsfromEvalTestinterface, all parsers, and test fixtures (~18 files)resolveInputWithFileContent()from llm-judge.ts (the patch becomes unnecessary)resolvedPathlives on the content segment or is derived at point of useAcceptance Signals
input_segmentsdoes not appear anywhere inpackages/core/src/orapps/cli/src/resolveInputWithFileContentis removed from llm-judge.tsbun run buildsucceedsbun test packages/core/test/evaluation/— all tests passagentv eval examples/features/multi-turn-conversation/evals/dataset.eval.yaml --target default— 2/2 pass with per-turn scoring and role-annotated context{{ input }}show resolved file content (not just paths)Non-Goals
TestMessagetype definition (content segments are alreadyJsonObject)processMessages)Related