Skip to content

Evaluate AGENTS.md and skills with Codex - #70120

Merged
jason810496 merged 2 commits into
apache:mainfrom
RoyLee1224:feat/skill-eval-codex-runtime
Aug 12, 2026
Merged

Evaluate AGENTS.md and skills with Codex#70120
jason810496 merged 2 commits into
apache:mainfrom
RoyLee1224:feat/skill-eval-codex-runtime

Conversation

@RoyLee1224

@RoyLee1224 RoyLee1224 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

cc: @jason810496

Follow up on #69308 by adding Codex as an alternative runtime for evaluating AGENTS.md and skills through promptfoo's built-in Codex SDK provider.

Run the evaluation with Codex:

AGENT_RUNTIME=codex \
  prek run run-skill-eval --hook-stage manual --all-files

The evaluation completed all six cases with no provider errors. Four behavioral assertions passed and two existing ambiguous newsfragment cases failed.

CleanShot 2026-07-20 at 17 26 29@2x CleanShot 2026-07-20 at 17 26 48@2x
Was generative AI tooling used to co-author this PR?
  • Yes gpt-5.6-sol

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Important

🛠️ Maintainer triage note for @RoyLee1224 · by @potiuk · 2026-07-28 16:32 UTC

Some review feedback from jason810496 is waiting on you:

  • There are 5 unresolved review threads on this PR from jason810496.

The ball is in your court — you've been assigned to this PR. Push a fix or reply in each thread explaining why the feedback does not apply, then mark them resolved and ping the reviewer (jason810496) for a final look.

See the Pull Request quality criteria for how to fix each item. There is no rush.

Note: your branch is 225 commits behind main — please rebase and push again to get up-to-date CI results.

Automated triage — may be imperfect; a maintainer takes the next look. We use this two-stage triage process so maintainers' limited time goes to the conversation with you.

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the Codex support!

Comment thread dev/skill-evals/eval.py Outdated
Comment thread .pre-commit-config.yaml
Comment thread .pre-commit-config.yaml
Comment thread dev/skill-evals/eval.py
Comment thread dev/skill-evals/eval.py
@RoyLee1224
RoyLee1224 force-pushed the feat/skill-eval-codex-runtime branch 2 times, most recently from 2efdf52 to f1d0ecc Compare July 30, 2026 07:47

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after resolving the following comments, thanks.

Comment thread .pre-commit-config.yaml
Comment thread scripts/tests/ci/prek/test_skill_eval.py Outdated
@RoyLee1224
RoyLee1224 force-pushed the feat/skill-eval-codex-runtime branch from f1d0ecc to 768733c Compare August 12, 2026 02:50

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@jason810496
jason810496 merged commit 2f5a376 into apache:main Aug 12, 2026
150 of 152 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants