Skip to content

ANS-1121: add optimizer judge model flag - #36

Open
jharris1679 wants to merge 2 commits into
mainfrom
josh/ans-1121-judge-model
Open

ANS-1121: add optimizer judge model flag#36
jharris1679 wants to merge 2 commits into
mainfrom
josh/ans-1121-judge-model

Conversation

@jharris1679

Copy link
Copy Markdown
Collaborator

Summary

  • add --judge-model as an explicit optimizer role override
  • make shared --model apply to eval, judge, trace analysis, and component author roles
  • preserve each eval case configured judge when neither shared nor explicit judge model is supplied
  • document the role behavior and add CLI request-contract coverage

Validation

  • npm test — 72 passed
  • git diff --check — passed

Companion to the answerlayer-core ANS-1121 implementation.

@greptile-apps

greptile-apps Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds optimizer model-role selection and documents its fallback behavior.

  • Adds --judge-model as an explicit judge override.
  • Applies the shared --model to eval, judge, trace-analysis, and proposal roles.
  • Preserves case-configured judges when neither a shared nor explicit judge model is supplied.
  • Adds request-contract coverage for explicit, shared, and omitted judge models.

Confidence Score: 5/5

The PR appears safe to merge with the model-role behavior correctly implemented, tested, and documented.

No actionable new issues remain, and the previous documentation finding is fully addressed by explicitly describing the analysis-model fallback order.

Important Files Changed

Filename Overview
src/cli.js Adds parsing, help text, and request serialization for the judge-model override.
test/cli.test.js Verifies explicit, shared, and omitted judge-model request behavior.
README.md Documents role overrides and the proposal-to-analysis fallback, resolving the previous finding.

Reviews (2): Last reviewed commit: "docs: clarify optimizer model precedence" | Re-trigger Greptile

Comment thread README.md Outdated
Comment on lines +252 to +255
The shared `--model` defaults the eval agent, judge, trace analyzer, and
component author. Use `--eval-model`, `--judge-model`, `--analysis-model`, or
`--proposal-model` for explicit role overrides. Without a shared model or
`--judge-model`, each case keeps its configured judge.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Analysis fallback is undocumented

The role description presents --proposal-model and --analysis-model as independent overrides, but the CLI resolves the analysis model from --analysis-model, then --proposal-model, then the shared --model. Without this detail, users may unintentionally select their proposal model for trace analysis.

Suggested change
The shared `--model` defaults the eval agent, judge, trace analyzer, and
component author. Use `--eval-model`, `--judge-model`, `--analysis-model`, or
`--proposal-model` for explicit role overrides. Without a shared model or
`--judge-model`, each case keeps its configured judge.
The shared `--model` defaults the eval agent, judge, trace analyzer, and
component author. Use `--eval-model`, `--judge-model`, `--analysis-model`, or
`--proposal-model` for explicit role overrides. Without `--analysis-model`,
trace analysis uses `--proposal-model` before falling back to the shared model.
Without a shared model or `--judge-model`, each case keeps its configured judge.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@jharris1679

Copy link
Copy Markdown
Collaborator Author

@greptileai review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant