Evaluate AGENTS.md and skills with Codex - #70120
Merged
jason810496 merged 2 commits intoAug 12, 2026
Merged
Conversation
RoyLee1224
requested review from
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jason810496,
jedcunningham,
jscheffl,
potiuk and
vatsrahul1001
as code owners
July 20, 2026 09:28
jason810496
reviewed
Jul 22, 2026
jason810496
left a comment
Member
There was a problem hiding this comment.
Thanks for adding the Codex support!
RoyLee1224
force-pushed
the
feat/skill-eval-codex-runtime
branch
2 times, most recently
from
July 30, 2026 07:47
2efdf52 to
f1d0ecc
Compare
jason810496
reviewed
Aug 9, 2026
jason810496
left a comment
Member
There was a problem hiding this comment.
LGTM after resolving the following comments, thanks.
RoyLee1224
force-pushed
the
feat/skill-eval-codex-runtime
branch
from
August 12, 2026 02:50
f1d0ecc to
768733c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cc: @jason810496
Follow up on #69308 by adding Codex as an alternative runtime for evaluating AGENTS.md and skills through promptfoo's built-in Codex SDK provider.
Run the evaluation with Codex:
The evaluation completed all six cases with no provider errors. Four behavioral assertions passed and two existing ambiguous newsfragment cases failed.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.Important
🛠️ Maintainer triage note for @RoyLee1224 · by
@potiuk· 2026-07-28 16:32 UTCSome review feedback from
jason810496is waiting on you:jason810496.The ball is in your court — you've been assigned to this PR. Push a fix or reply in each thread explaining why the feedback does not apply, then mark them resolved and ping the reviewer (
jason810496) for a final look.See the Pull Request quality criteria for how to fix each item. There is no rush.
Note: your branch is 225 commits behind
main— please rebase and push again to get up-to-date CI results.Automated triage — may be imperfect; a maintainer takes the next look. We use this two-stage triage process so maintainers' limited time goes to the conversation with you.