feat(proforge): plan Claude structured-outputs wiring - #730
Conversation
🤖 CodeAnt AI — Review Status
|
Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reachedNext included review available in 55 minutes. View limit detailsLimit details: You’ve used the included review currently available. Your 72 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Essentials Run ID: 📒 Files selected for processing (1)
Comment |
Reviewer's GuideThis documentation-only PR captures the implementation plan for wiring Anthropic native structured outputs into ProForge: it identifies the six schema-backed JSON stages, records the API's restrictive schema requirements, evaluates sanitizer versus SDK conversion, and maps the request plumbing and offline verification steps needed for a future implementation. Sequence diagram for planned Anthropic structured-output requestsequenceDiagram
participant Agent as ProForge agent
participant Base as baseAgent.generate
participant Service as aiProviderService
participant Proxy as claudeProxyCore
participant Claude as Claude Messages API
participant Validator as validateWithSchema
Agent->>Base: generate(prompt, model, responseSchema)
Base->>Service: generateText(options)
Service->>Proxy: streamAnthropic(responseSchema)
Proxy->>Claude: Messages API with output_config.format
Claude-->>Proxy: text content block containing JSON
Proxy-->>Service: Anthropic response
Service-->>Agent: response text
Agent->>Validator: validateWithSchema(response)
Flow diagram for Anthropic schema conversion and verificationflowchart TD
Zod[Zod schema] --> JSONSchema[z.toJSONSchema]
JSONSchema --> Sanitizer[toAnthropicJsonSchema]
Sanitizer --> Wire[Anthropic-compatible JSON schema]
Wire --> Checks[Throwaway schema-shape checks]
Checks --> Typecheck[pnpm run typecheck]
Checks --> Lint[pnpm run lint]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
🏁 CodeAnt Quality Gate ResultsCommit: ✅ Overall Status: PASSEDQuality Gate Details
|
There was a problem hiding this comment.
No application code in the PR — skipped Code Health checks.
See analysis details in CodeScene
Quality Gate Profile: The Bare Minimum
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.
| 2. `services/aiProviderService.ts` — add `responseSchema?: Record<string, unknown>` to `AIRequestOptions`; thread it into **both** `streamAnthropic` branches (the Tauri-direct desktop path at `:397-413` and the proxy path at `:415-427`). No other provider reads this field — safe no-op for OpenAI/Grok/Gemini/Ollama/OpenRouter/local. | ||
| 3. `services/proForge/pipelineAgents/baseAgent.ts` — `generate()` (`:117-130`) gains an optional third `responseSchema` param, forwarded into `buildAiOpts()`. `InferenceGateway.generate()` (`services/ai/inferenceGateway.ts`) needs **no changes** — it already forwards `options: AIRequestOptions` wholesale to `generateText()`. | ||
| 4. New helper, e.g. `services/proForge/pipelineOutput/anthropicJsonSchema.ts`, exporting `toAnthropicJsonSchema(schema: z.ZodType): Record<string, unknown>` implementing option (a) above if chosen. | ||
| 5. Six call sites — pass the converted schema as the third arg: `diagnosticAgent.ts` (`diagnosticReportSchema`), `structuralAgent.ts` (`structuralEditPlanSchema`), `proseAgent.ts` (`proseEditBatchSchema`, per-section), `copyEditAgent.ts` (`copyEditPlanSchema`, per-section), `proofAgent.ts` (`qualityGateReportSchema`), `publishingAgent.ts` (`publishingPackageSchema`). |
There was a problem hiding this comment.
Suggestion: The six named agents contain eight generate invocation paths because diagnostic and structural retries call it again; omitting those retries leaves them unconstrained. [incomplete implementation]
Assessment: 🟠 Major · 🔁 Occurrence: Sometimes
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** docs/PROFORGE-CLAUDE-STRUCTURED-OUTPUTS-PLAN.md
**Line:** 55:55
**Comment:**
*Incomplete Implementation: The six named agents contain eight `generate` invocation paths because diagnostic and structural retries call it again; omitting those retries leaves them unconstrained.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
Execution owner / issue linkageThis prep-only PR is the planning/evidence artifact for #729 — Canonical relationship: #730 documents the verified current API/schema constraints and implementation path; it does not close #729 and should not be treated as runtime implementation. Sibling audit tracks:
When implementation begins, re-fetch current |
There was a problem hiding this comment.
1 issue found across 1 file
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="docs/PROFORGE-CLAUDE-STRUCTURED-OUTPUTS-PLAN.md">
<violation number="1" location="docs/PROFORGE-CLAUDE-STRUCTURED-OUTPUTS-PLAN.md:55">
P2: Include the diagnostic and structural retry `generate` paths in this plumbing step. If only the six listed calls receive `responseSchema`, those retries continue using prose JSON and bypass the native structured-output contract.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| 2. `services/aiProviderService.ts` — add `responseSchema?: Record<string, unknown>` to `AIRequestOptions`; thread it into **both** `streamAnthropic` branches (the Tauri-direct desktop path at `:397-413` and the proxy path at `:415-427`). No other provider reads this field — safe no-op for OpenAI/Grok/Gemini/Ollama/OpenRouter/local. | ||
| 3. `services/proForge/pipelineAgents/baseAgent.ts` — `generate()` (`:117-130`) gains an optional third `responseSchema` param, forwarded into `buildAiOpts()`. `InferenceGateway.generate()` (`services/ai/inferenceGateway.ts`) needs **no changes** — it already forwards `options: AIRequestOptions` wholesale to `generateText()`. | ||
| 4. New helper, e.g. `services/proForge/pipelineOutput/anthropicJsonSchema.ts`, exporting `toAnthropicJsonSchema(schema: z.ZodType): Record<string, unknown>` implementing option (a) above if chosen. | ||
| 5. Six call sites — pass the converted schema as the third arg: `diagnosticAgent.ts` (`diagnosticReportSchema`), `structuralAgent.ts` (`structuralEditPlanSchema`), `proseAgent.ts` (`proseEditBatchSchema`, per-section), `copyEditAgent.ts` (`copyEditPlanSchema`, per-section), `proofAgent.ts` (`qualityGateReportSchema`), `publishingAgent.ts` (`publishingPackageSchema`). |
There was a problem hiding this comment.
P2: Include the diagnostic and structural retry generate paths in this plumbing step. If only the six listed calls receive responseSchema, those retries continue using prose JSON and bypass the native structured-output contract.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/PROFORGE-CLAUDE-STRUCTURED-OUTPUTS-PLAN.md, line 55:
<comment>Include the diagnostic and structural retry `generate` paths in this plumbing step. If only the six listed calls receive `responseSchema`, those retries continue using prose JSON and bypass the native structured-output contract.</comment>
<file context>
@@ -0,0 +1,63 @@
+2. `services/aiProviderService.ts` — add `responseSchema?: Record<string, unknown>` to `AIRequestOptions`; thread it into **both** `streamAnthropic` branches (the Tauri-direct desktop path at `:397-413` and the proxy path at `:415-427`). No other provider reads this field — safe no-op for OpenAI/Grok/Gemini/Ollama/OpenRouter/local.
+3. `services/proForge/pipelineAgents/baseAgent.ts` — `generate()` (`:117-130`) gains an optional third `responseSchema` param, forwarded into `buildAiOpts()`. `InferenceGateway.generate()` (`services/ai/inferenceGateway.ts`) needs **no changes** — it already forwards `options: AIRequestOptions` wholesale to `generateText()`.
+4. New helper, e.g. `services/proForge/pipelineOutput/anthropicJsonSchema.ts`, exporting `toAnthropicJsonSchema(schema: z.ZodType): Record<string, unknown>` implementing option (a) above if chosen.
+5. Six call sites — pass the converted schema as the third arg: `diagnosticAgent.ts` (`diagnosticReportSchema`), `structuralAgent.ts` (`structuralEditPlanSchema`), `proseAgent.ts` (`proseEditBatchSchema`, per-section), `copyEditAgent.ts` (`copyEditPlanSchema`, per-section), `proofAgent.ts` (`qualityGateReportSchema`), `publishingAgent.ts` (`publishingPackageSchema`).
+6. Leave the "Return JSON with…" prose in `promptLibrary.ts` untouched — it's still load-bearing for every non-Anthropic provider (Gemini has its own native JSON path already; OpenAI/Grok/Ollama/OpenRouter/local all still need the prose instruction). Redundant-but-harmless for Claude once structured outputs are wired.
+
</file context>
| 5. Six call sites — pass the converted schema as the third arg: `diagnosticAgent.ts` (`diagnosticReportSchema`), `structuralAgent.ts` (`structuralEditPlanSchema`), `proseAgent.ts` (`proseEditBatchSchema`, per-section), `copyEditAgent.ts` (`copyEditPlanSchema`, per-section), `proofAgent.ts` (`qualityGateReportSchema`), `publishingAgent.ts` (`publishingPackageSchema`). | |
| 5. Eight `generate` invocation paths across the six agents — pass the converted schema as the third arg on every initial and diagnostic/structural retry call: `diagnosticAgent.ts` (`diagnosticReportSchema`), `structuralAgent.ts` (`structuralEditPlanSchema`), `proseAgent.ts` (`proseEditBatchSchema`, per-section), `copyEditAgent.ts` (`copyEditPlanSchema`, per-section), `proofAgent.ts` (`qualityGateReportSchema`), `publishingAgent.ts` (`publishingPackageSchema`). |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 953a1feec0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| 1. Implement the chosen sanitizer/dependency approach. | ||
| 2. Write a throwaway script (`npx tsx` one-shot, not a committed file) that runs `toAnthropicJsonSchema()` against all six schemas and asserts: no `minimum`/`maximum`/`exclusiveMinimum`/`exclusiveMaximum`/`multipleOf`/`minLength`/`maxLength` keys anywhere in the output, no `maxItems`/`minItems` > 1, and `additionalProperties: false` present on every object node (including inside `$defs`). This is checkable without live API access. | ||
| 3. `pnpm run typecheck` + `pnpm run lint`. | ||
| 4. If a real Anthropic API key is available, smoke-test one stage (e.g. `DiagnosticAgent`) end-to-end and confirm the response still parses via the existing `validateWithSchema` safety net. |
There was a problem hiding this comment.
When this plan is implemented, the throwaway schema assertion, typecheck, lint, and optional live smoke test will not verify that output_config.format is forwarded correctly—or omitted for ordinary requests—through both the desktop and proxy branches. Add committed focused coverage for the sanitizer and extend tests/unit/aiProviderService.test.ts and tests/unit/api/claudeProxyCore.test.ts; these are the existing narrow seams for preventing regressions in this non-trivial network behavior.
AGENTS.md reference: AGENTS.md:L16-L20
Useful? React with 👍 / 👎.
| ## Verification plan for next session | ||
|
|
||
| 1. Implement the chosen sanitizer/dependency approach. | ||
| 2. Write a throwaway script (`npx tsx` one-shot, not a committed file) that runs `toAnthropicJsonSchema()` against all six schemas and asserts: no `minimum`/`maximum`/`exclusiveMinimum`/`exclusiveMaximum`/`multipleOf`/`minLength`/`maxLength` keys anywhere in the output, no `maxItems`/`minItems` > 1, and `additionalProperties: false` present on every object node (including inside `$defs`). This is checkable without live API access. |
There was a problem hiding this comment.
A future implementer following this command will invoke npm's package executor even though tsx is already a declared repository dependency and local dependency operations must use pnpm. Use pnpm exec tsx so execution stays within the pinned pnpm dependency graph.
AGENTS.md reference: AGENTS.md:L11-L12
Useful? React with 👍 / 👎.
| The skill's recommended path (`zodOutputFormat(schema)` from `@anthropic-ai/sdk/helpers/zod`) auto-strips the unsupported keywords and is the officially-blessed approach — but this repo has **no** `@anthropic-ai/sdk` dependency anywhere; it deliberately uses raw `fetch` for the whole Anthropic path (browser bundle + the edge proxy) to stay dependency-light and avoid CORS/bundle-size issues. Pulling in the full Node SDK just for one schema-conversion helper is a heavy tradeoff for a client bundle. Decide this explicitly next session — it's the one open design choice here — between: | ||
|
|
||
| - **(a)** write a small local sanitizer (recursive walk over `z.toJSONSchema()`'s output: drop `minimum`/`maximum`/`exclusiveMinimum`/`exclusiveMaximum`/`multipleOf`/`minLength`/`maxLength`, drop `maxItems`/`minItems` > 1, force `additionalProperties: false` onto every `type: "object"` node including inside `$defs`), or | ||
| - **(b)** add `@anthropic-ai/sdk` (server-only, e.g. scoped to the edge function if bundling allows) and use `zodOutputFormat()` directly. |
There was a problem hiding this comment.
If option (b) is selected as written and @anthropic-ai/sdk is scoped only to the edge function, the Tauri-direct branch cannot produce the sanitized schema: services/aiProviderService.ts sends desktop requests straight to api.anthropic.com, while the proposed call sites can only pass serializable JSON, not a Zod schema for the edge to convert. Either make the converter client-compatible for both branches or define separate proxy and desktop conversion plumbing before presenting server-only placement as a viable option.
Useful? React with 👍 / 👎.
| 2. `services/aiProviderService.ts` — add `responseSchema?: Record<string, unknown>` to `AIRequestOptions`; thread it into **both** `streamAnthropic` branches (the Tauri-direct desktop path at `:397-413` and the proxy path at `:415-427`). No other provider reads this field — safe no-op for OpenAI/Grok/Gemini/Ollama/OpenRouter/local. | ||
| 3. `services/proForge/pipelineAgents/baseAgent.ts` — `generate()` (`:117-130`) gains an optional third `responseSchema` param, forwarded into `buildAiOpts()`. `InferenceGateway.generate()` (`services/ai/inferenceGateway.ts`) needs **no changes** — it already forwards `options: AIRequestOptions` wholesale to `generateText()`. | ||
| 4. New helper, e.g. `services/proForge/pipelineOutput/anthropicJsonSchema.ts`, exporting `toAnthropicJsonSchema(schema: z.ZodType): Record<string, unknown>` implementing option (a) above if chosen. | ||
| 5. Six call sites — pass the converted schema as the third arg: `diagnosticAgent.ts` (`diagnosticReportSchema`), `structuralAgent.ts` (`structuralEditPlanSchema`), `proseAgent.ts` (`proseEditBatchSchema`, per-section), `copyEditAgent.ts` (`copyEditPlanSchema`, per-section), `proofAgent.ts` (`qualityGateReportSchema`), `publishingAgent.ts` (`publishingPackageSchema`). |
There was a problem hiding this comment.
There are eight structured-output invocations rather than six: DiagnosticAgent and StructuralAgent each call generate() again after an incoherent self-reflection (diagnosticAgent.ts:89 and structuralAgent.ts:82). If the implementation follows this six-call-site instruction literally, those retries will omit the schema and can again return malformed prose JSON precisely when the first response needed correction; explicitly include both retry invocations while leaving selfReflect() itself text-only.
Useful? React with 👍 / 👎.
| @@ -0,0 +1,63 @@ | |||
| # Plan: wire Claude's native structured outputs into ProForge | |||
|
|
|||
| Status: **planned, not yet implemented** — prep doc from a `/claude-api prompt-audit` pass (2026-09-12). Implement in this branch (`feat/proforge-claude-structured-outputs`) when work resumes. This is the largest of the three audit follow-ups; the other two (`docs/PROFORGE-CLAUDE-TOKEN-ACCOUNTING-PLAN.md`, `docs/PROFORGE-CLAUDE-MAXTOKENS-CEILING-PLAN.md`) are independent and smaller. | |||
There was a problem hiding this comment.
The reviewed commit presents two concrete documentation paths as the other audit follow-ups, but a repo-wide search and target-tree listing find neither file or any other reference to them. Readers therefore cannot inspect the supposedly completed audit split; either include these plans, link to existing tracked work, or state explicitly that the named files have not yet been created.
Useful? React with 👍 / 👎.
|
|
||
| The skill's recommended path (`zodOutputFormat(schema)` from `@anthropic-ai/sdk/helpers/zod`) auto-strips the unsupported keywords and is the officially-blessed approach — but this repo has **no** `@anthropic-ai/sdk` dependency anywhere; it deliberately uses raw `fetch` for the whole Anthropic path (browser bundle + the edge proxy) to stay dependency-light and avoid CORS/bundle-size issues. Pulling in the full Node SDK just for one schema-conversion helper is a heavy tradeoff for a client bundle. Decide this explicitly next session — it's the one open design choice here — between: | ||
|
|
||
| - **(a)** write a small local sanitizer (recursive walk over `z.toJSONSchema()`'s output: drop `minimum`/`maximum`/`exclusiveMinimum`/`exclusiveMaximum`/`multipleOf`/`minLength`/`maxLength`, drop `maxItems`/`minItems` > 1, force `additionalProperties: false` onto every `type: "object"` node including inside `$defs`), or |
There was a problem hiding this comment.
Under option (a), simply deleting the unsupported bounds makes the Anthropic schema weaker than the Zod validator that still consumes the response. Several constraints are not stated in the prompts—for example the publishing blurb length limits and multiple prose score ranges—so Claude can return JSON that satisfies the sanitized schema but fails validateWithSchema, causing an otherwise valid report, section batch, or publishing package to be discarded or replaced by a fallback. Translate removed bounds into supported descriptions or explicit prompt guidance rather than silently dropping their semantics.
Useful? React with 👍 / 👎.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
User description
Summary
/claude-api prompt-auditpass — captures the largest of the three audit follow-ups so next week's session can implement directly instead of re-auditing or re-researching the API.stripJsonFences+ a regex brace-match fallback — even though the app already owns complete Zod schemas for every one of those shapes.docs/PROFORGE-CLAUDE-STRUCTURED-OUTPUTS-PLAN.md— the verified live wire shape and hard constraints ofoutput_config.format(fetched from Anthropic's current docs, 2026-09-12:additionalProperties: falserequired everywhere,minimum/maximum/minLength/maxLength/maxItems>1all rejected with a 400 — which every existing schema in this repo currently uses), the one open design decision (small local schema sanitizer vs. adding@anthropic-ai/sdkjust forzodOutputFormat()), the full plumbing path file-by-file, and a verification plan that doesn't require live API access.Test plan
pnpm run typecheck+pnpm run lint.Summary by Sourcery
Document the implementation plan for adopting Claude structured outputs across ProForge without changing runtime behavior.
Enhancements:
Documentation:
Summary by cubic
Adds
docs/PROFORGE-CLAUDE-STRUCTURED-OUTPUTS-PLAN.md, a prep plan from the Claude API audit, so the structured-outputs wiring can be implemented in one session without re-research.output_config.format, including constraints that break every existing schema (additionalProperties: falserequired;minimum,maxLength,maxItemsover 1 all rejected).@anthropic-ai/sdkforzodOutputFormat()— plus the file-by-file plumbing path and a verification plan that needs no live API access.No code changes; this PR is docs only.
Written for commit 953a1fe. Summary will update on new commits.
CodeAnt-AI Description
Document the planned migration of ProForge's JSON-producing stages to Claude's native structured outputs
What Changed
Impact
✅ Clear implementation path for structured Claude responses✅ Fewer unexpected Claude schema request failures✅ Existing response validation remains protected💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.