Motivation
Link to the gh-aw PR: github/gh-aw#47956 — propagate ModelMappings into evals WorkflowData to unblock AWF model alias resolution.
Evals jobs were producing 100% UNKNOWN answers because buildEvalsEngineSteps constructed evalsData without copying ModelMappings from the parent WorkflowData. This caused the AWF to exit with code 1 on an unresolvable model alias (e.g. "small"), silently discarded by continue-on-error: true. There is currently no E2E test that exercises the evals: frontmatter section at all, leaving this entire feature path untested.
Proposed test
- Workflow file:
test-copilot-evals-frontmatter.md
- Trigger:
workflow_dispatch
- Engine: copilot
- Safe output:
create-issue (the eval result is checked via the evals artifact, not a safe output)
- Variant: standard
Minimal test prompt sketch
A minimal workflow with an evals: block in the frontmatter that uses a model alias (e.g. model: small). The agent is asked a fixed factual question. The test asserts that the evals job produces an evals.jsonl artifact (i.e. exit 0, not the silent failure from the alias-resolution bug) and that the overall workflow run succeeds.
New fixtures or secrets needed
None beyond existing engine credentials. The evals: feature uses the standard Copilot engine path.
Notes
There is an open suggestion Suggested test: copilot/evals-frontmatter already in open-suggestions.json. This issue provides the specific motivation from PR #47956 and the model-alias regression angle. If the earlier suggestion is already being acted on, this can be closed as a duplicate — but the model-alias regression angle (using model: small) should be incorporated into any evals test.
Generated by 🔍 Suggest New E2E Tests · sonnet46 · 32.7 AIC · ⌖ 8.96 AIC · ⊞ 6.1K · ◷
Motivation
Link to the gh-aw PR: github/gh-aw#47956 — propagate
ModelMappingsinto evalsWorkflowDatato unblock AWF model alias resolution.Evals jobs were producing 100%
UNKNOWNanswers becausebuildEvalsEngineStepsconstructedevalsDatawithout copyingModelMappingsfrom the parentWorkflowData. This caused the AWF to exit with code 1 on an unresolvable model alias (e.g."small"), silently discarded bycontinue-on-error: true. There is currently no E2E test that exercises theevals:frontmatter section at all, leaving this entire feature path untested.Proposed test
test-copilot-evals-frontmatter.mdworkflow_dispatchcreate-issue(the eval result is checked via the evals artifact, not a safe output)Minimal test prompt sketch
A minimal workflow with an
evals:block in the frontmatter that uses a model alias (e.g.model: small). The agent is asked a fixed factual question. The test asserts that the evals job produces anevals.jsonlartifact (i.e. exit 0, not the silent failure from the alias-resolution bug) and that the overall workflow run succeeds.New fixtures or secrets needed
None beyond existing engine credentials. The
evals:feature uses the standard Copilot engine path.Notes
There is an open suggestion
Suggested test: copilot/evals-frontmatteralready inopen-suggestions.json. This issue provides the specific motivation from PR #47956 and the model-alias regression angle. If the earlier suggestion is already being acted on, this can be closed as a duplicate — but the model-alias regression angle (usingmodel: small) should be incorporated into any evals test.