Skip to content

[DEV-107] Cleaner AI-generated task content by using harness structured output mode - #100

Open
danii1 wants to merge 6 commits into
mainfrom
feature/dev-107
Open

[DEV-107] Cleaner AI-generated task content by using harness structured output mode#100
danii1 wants to merge 6 commits into
mainfrom
feature/dev-107

Conversation

@danii1

@danii1 danii1 commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Task: DEV-107

Summary: Cleaner AI-generated task content by using harness structured output mode

Implementation Details

I'll start by exploring the codebase to understand the existing structures before implementing.
Now let me look at the pm engine:
Let me look at the remaining key files — agent-json.ts, engine types, tests, and how pm-desktop consumes onAgentChunk:
Now let me check how pm-desktop consumes chunks and how config is structured:
Let me look at the config structure, the prompts, and how the chat bot uses the engine:
Let me check the envelope shapes documented in harness sources, and the chunk consumers:
I now have full context. Let me set up the task plan:
Now creating the structured-output engine module:
Now wiring the engine (packages/pm/lib/engine/index.ts):
Now the runAndParse core change:
Now add the env opt-out helper to structured.ts:
Core wiring is complete. Now let me add tests to packages/pm/engine.test.ts:
Now add the structured-output test suites before the engine module isolation block:
These look like missing dependency types. Let me install deps first:
Typecheck passes. Now run the pm tests:
Found it — a regression: when structured mode is off but onAgentChunk exists, stdout chunks were dropped. Fixing:
All pm tests pass. Now update todos and docs:
Now run the full verification suite:
All builds pass. Let me run the root-level checks to make sure nothing else broke:
All monorepo checks pass. Final review of the diff:
DEV-107 is implemented. Summary of the change:

New module — packages/pm/lib/engine/structured.ts

  • unwrapStructuredPayload: unwraps harness envelopes so isStoryPayload/isDecompositionPayload keep working — Claude/Cursor/Grok/Goose/Antigravity {"type":"result",...,"result":"<reply>"} single-doc envelopes, Codex/Opencode/Kilo/Cline/Kimi/Pi NDJSON event arrays (last text-bearing entry, incl. nested item.text), and Qwen buffered message arrays.
  • structuredPayloadFromResult: reads result.structured; when parsing failed (ok === false) or the unwrapped payload doesn't validate, returns not-ok so the engine falls back to extractJsonPayload on raw stdout. Reply text that isn't clean JSON still gets the tolerant repair.
  • createReadableStdoutTap: in JSON mode, rewrites JSON event stdout lines carrying assistant text into readable text for live output views (pm-desktop agent log), passing everything else through raw; plain-text runs forward chunks byte-for-byte as before.
  • AGENT_STRUCTURED_OUTPUT env opt-out (0/false/off/no) for harness-specific JSON-mode quirks.

Engine wiring — packages/pm/lib/engine/index.ts

  • runAndParse requests structuredOutput: true only when the configured harness declares supportsStructuredOutput (capability-gated; non-supporting harnesses keep the untouched plain-text path and can never see UnsupportedStructuredOutputError).
  • Reads the payload from result.structured first; falls back to the existing extractJsonPayload repair on raw stdout; the one-shot STRICT_JSON_REMINDER corrective re-run still applies on the fallback path. Transcript detectors and debug dumps are untouched (they scan raw streams in the runner).

pm-desktop needs no code change — it builds the engine via createEngine (src/main/session.ts:481), so it gets structured mode and readable streaming automatically across multi-ticket/edit flows.

Tests — 14 new cases in packages/pm/engine.test.ts: JSON flag emitted for a supporting harness, plain-path fallback for non-supporting harnesses, fallback when structured.ok === false or payload fails validation, envelope/NDJSON/message-array unwrapping, narration repair inside reply text, corrective re-run intact, both opt-outs, and stream-tap behavior (readable text, raw passthrough, flush, stderr untouched).

Docs — docs/pm/configuration.md gains a "Structured (JSON) output" section documenting the behavior and AGENT_STRUCTURED_OUTPUT.

Verification: bun run typecheck, lint, test, and build all pass across the monorepo (pm: 352 tests, pm-desktop: 543, agent-harness unchanged).

This PR was automatically created by @devintern/code

@danii1 danii1 added the devintern-authored Pull request written by DevIntern working its own backlog label Aug 29, 2026
danii1
danii1 commented Sep 2, 2026
View reviewed changes
Comment thread docs/pm/configuration.md Outdated

```bash
# .devintern-pm/.env
AGENT_STRUCTURED_OUTPUT=0

@danii1 danii1 Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@devintern-ai if structured output is supported by harness let's treat that it outputs info correctly and engine should use it, we won't offer overrides by the env vars as it's our problem to verify that every harness works correctly in this mode

Comment thread docs/pm/configuration.md Outdated

In the DevIntern PM desktop app, set the same override per project from **Settings → Agent model**; it persists to `.devintern-pm/.env` (same file the CLI reads) and applies to new agent runs immediately.

### Structured (JSON) output

@danii1 danii1 Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@devintern-ai let's not modify docs/pm/configuration.md as it's our internals and for user it doesn't matter

@danii1
refactor: drop AGENT_STRUCTURED_OUTPUT env override and revert intern…
c1058d8 
…al docs (review feedback)

- Revert docs/pm/configuration.md: structured output is internal engine
  behavior, not user-facing configuration
- Remove AGENT_STRUCTURED_OUTPUT opt-out env var and the createEngine
  structuredOutput override; harnesses supporting JSON mode are always
  asked for structured output and its parsed payload is trusted
- Drop the corresponding opt-out tests
danii1
danii1 commented Sep 2, 2026
View reviewed changes
Comment thread packages/pm/lib/engine/structured.ts Outdated
* {@link MAX_REPLY_TEXT_DEPTH}. Returns `undefined` when nothing readable is
* found.
*/
function extractReplyText(value: unknown, depth: number): string | undefined {

@danii1 danii1 Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@devintern-ai you need to pass a structured output schema or use the exact harness schema(they differ from harness to harness) to parse the output properly. Exact harness schema maybe preferrable as it lets us surface stats like token usage and costs

@danii1
refactor: parse structured output via exact per-harness envelope sche…
7ec67c3 
…mas (DEV-107 review)

Address review feedback on packages/pm/lib/engine/structured.ts: the generic
reply-text field sniffing (REPLY_TEXT_FIELDS / extractReplyText) could not
parse harness envelopes reliably, since each harness's JSON mode has its own
shape. Implement the reviewer-preferred option: exact harness schemas, which
also unlock usage/cost stats.

- agent-harness: new src/structured-envelope.ts registers an exact envelope
  schema per structured-output harness (claude-code result envelope, codex
  item/turn events, opencode step/text events, qwen result entries, kimi
  messages, cline/kilo say events, pi session events, cursor/grok/deepseek
  result and goose/antigravity response envelopes). extractHarnessStructuredReply
  recovers the model's final reply plus normalized token usage and cost;
  extractHarnessEventText maps single event lines to readable text. Unknown
  harnesses/mismatches yield nothing so callers fail open to legacy handling.
- pm engine: structuredPayloadFromResult and createReadableStdoutTap are keyed
  on the configured harness name; envelope field guessing is removed.
- pm engine: new EngineCallEvents.onAgentUsage surfaces token usage and cost
  reported by the harness envelope (e.g. Claude Code usage/total_cost_usd,
  Codex turn.completed, opencode step_finish).
- tests: agent-harness envelope schema suite; pm engine tests updated to
  harness-keyed fixtures plus usage-surfacing coverage; readme documents the
  per-harness schema table and adds a checklist step for new harnesses.
danii1
danii1 commented Sep 2, 2026
View reviewed changes
Comment thread packages/agent-harness/src/structured-envelope.ts
@danii1
fix: parse grok structured output from its verified envelope shape (D…
00f10a7 
…EV-107 review)

Reviewer reported a live grok --output-format json run failing to parse:
the envelope carries the reply in `text` (not `result`), and the model
wraps the JSON payload in narration prose.

Verified the exact shape against the open-source CLI (xai-org/grok-build,
crates/codegen/xai-grok-pager/src/headless.rs `build_json_result`) and:

- register a dedicated grok envelope schema: reply text from `text`,
  Claude-style usage (incl. `reasoning_tokens`/`total_tokens`) plus
  `total_cost_usd` via a shared claudeStyleStats helper
- make parseReplyText recover JSON embedded in narration: fenced
  ```json blocks and string-aware balanced `{...}` spans are tried
  before falling back to the raw reply text (all harnesses benefit)
- pin the shape in the module header, harness comment, and readme
  envelope table; note the open-source-CLI verification path from the
  review discussion
@danii1
Merge remote-tracking branch 'origin/main' into feature/dev-107
8a3e8dd 
# Conflicts:
#	packages/agent-harness/tests/sandbox-providers.test.ts
@danii1

danii1 commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

🔀 This branch had merge conflicts with main; devintern resolved them and pushed the merge. Please double-check the resolution.

@danii1

danii1 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

@devintern-ai for opensource harnesses, check their repos and document/link their exact json response shape

@danii1
docs(agent-harness): verify and link exact upstream JSON shapes per h…
b5815e5 
…arness (DEV-107 review)

Checked each open-source harness's repo for its exact structured-output
JSON shape and documented it in the structured-envelope module header and
readme (with upstream repo + file links). Verification corrected three
schemas that did not match upstream:

- goose: `--output-format json` emits a `{messages, metadata}` document
  (reply = last assistant message's text blocks, usage/cost in
  `metadata`), not a `response` envelope
- cline: `--json` ends in a final `run_result` record (`text` +
  camelCase `usage`), not `say:"text"` records
- kilo-code: the CLI is an opencode fork, so it shares the opencode
  event schema instead of cline's

Also aligned parsers with the verified shapes: codex (no upstream
`total_tokens`; map `cache_write_input_tokens` /
`reasoning_output_tokens`), opencode (no `tokens.total`), pi (usage also
read from `message_end.message.usage`; `totalTokens`/`reasoning`/
`cost.total`), and deepseek (verified Claude-style usage +
`total_cost_usd`). Envelope tests now use upstream-shaped fixtures;
closed-source CLIs (claude-code, cursor, antigravity) are documented
from their official docs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

devintern-authored Pull request written by DevIntern working its own backlog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@danii1