Skip to content

Validate paired evaluation models on Pydantic completion - #316

Closed
1st1 wants to merge 2 commits into
evaluate-magic-genericsfrom
evaluation-model-validation
Closed

1st1 wants to merge 2 commits into
evaluate-magic-genericsfrom
evaluation-model-validation

Conversation

@1st1

@1st1 1st1 commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #307, based on its branch.

Use plain Pydantic BaseModel for answers. BaseQuestionModel stores __answers_type__ and validates both models in __pydantic_on_complete__, rebuilding the answer model before inspecting its fields. Supports resolved forward references and answer subclasses, and reports the exact fields missing from either model. Updates examples, docs, and tests.

Validation: 45 relevant tests pass, including explicit and automatic forward-reference rebuilding; Ruff format/lint, mypy, and ty pass.

@vercel

vercel Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
ai-python Ready Ready Preview, v0 Sep 28, 2026 9:16pm UTC

@1st1 1st1 changed the title Validate paired evaluation models at class creation Validate paired evaluation models on Pydantic completion Sep 28, 2026
Comment thread src/ai/ops/experimental/evaluation.py Outdated
):
raise TypeError(
"questions must specialize BaseQuestionModel "
"with a Pydantic model"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add an example here, like class Questions(ai.ops.experimental.BaseQuestionsModel[AnswerT]) ?

This branch was successfully deployed

1 active deployment
Preview — ccf0bcc7 Deployed Sep 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants