Skip to content

Rewrite experiments.<name> in engine.model to valid job-scoped expressions - #61599

Merged
pelikhan merged 12 commits into
mainfrom
copilot/fix-claude-engine-experiments-model
Sep 17, 2026
Merged

pelikhan merged 12 commits into
mainfrom
copilot/fix-claude-engine-experiments-model

Conversation

Copilot AI commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

${{ experiments.model }} used in engine.model (or the split top-level model: field) was rewritten for prompt-body text but not for ANTHROPIC_MODEL, GH_AW_INFO_MODEL, or GH_AW_ENGINE_MODEL. The compiled lock file kept the literal experiments.model reference, which GitHub Actions rejects at run time since experiments is not a valid context:

ANTHROPIC_MODEL: ${{ experiments.model }}   # invalid — not a GH Actions context

Fix

  • Added regex-based rewrite helpers (pkg/workflow/compiler_experiments.go) that convert experiments.<name> into a valid, job-scoped reference:
    • RewriteExperimentsReferenceForDownstreamJobs → needs.activation.outputs.<name>, for jobs downstream of activation (agent, detection, conclusion, safe-outputs).
    • RewriteExperimentsReferenceForActivationJob / RewriteActivationOutputsToLocalStepOutputs → steps.pick-experiment.outputs.<name>, for the activation job's own info step (a job cannot reference its own outputs via needs).
  • Applied RewriteExperimentsReferenceForDownstreamJobs to workflowData.Model right after experiments are extracted from frontmatter, so every consumer of data.Model (Claude/Codex/Copilot/Gemini/Pi engines' native model env vars, GH_AW_ENGINE_MODEL, etc.) picks up the corrected value automatically.
  • Applied RewriteActivationOutputsToLocalStepOutputs in generateCreateAwInfo so GH_AW_INFO_MODEL (emitted inside the activation job itself) uses the step-local form instead.
  • Matching is word-boundary-anchored and scoped strictly to declared experiment names, so an experiment name that is a prefix of another (e.g. model vs model_variant) can't corrupt an unrelated occurrence, and unrelated text is never rewritten.

Result

ANTHROPIC_MODEL: ${{ needs.activation.outputs.model }}      # agent job
GH_AW_INFO_MODEL: "${{ steps.pick-experiment.outputs.model }}"  # activation job
GH_AW_ENGINE_MODEL: "${{ needs.activation.outputs.model }}"     # conclusion job

Tests

  • Unit tests for each rewrite helper, including a name-prefix-collision regression case.
  • End-to-end compile tests asserting the lock file for a Claude workflow using experiments.model never contains a raw experiments.model token, for both the nested engine.model and split top-level model: forms.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 28.8 AIC · ⊞ 9.6K · ◷
Comment /souschef to run again


Run: https://github.com/github/gh-aw/actions/runs/35240793612

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 24.9 AIC · ⊞ 9.3K · ◷
Comment /souschef to run again

Copilot AI and others added 8 commits September 17, 2026 13:12
…sions

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…ed experiment names

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…prefix collisions

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix Claude engine: ${{ experiments.model }} not rewritten Rewrite experiments.<name> in engine.model to valid job-scoped expressions Sep 17, 2026
Copilot AI requested a review from pelikhan September 17, 2026 13:29
@pelikhan
pelikhan marked this pull request as ready for review September 17, 2026 13:30
Copilot AI balanced review requested due to automatic review settings September 17, 2026 13:30
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • ab.chatgpt.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "ab.chatgpt.com"

See Network Configuration for more information.

Generated by Ponytail Reviewer for #61599

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Security scanning failed for Design Decision Gate 🏗️. Review the logs for details.

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • github.com
  • proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"
    - "proxy.golang.org"

See Network Configuration for more information.

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ PR Code Quality Reviewer failed during code quality review.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer failed during the skills-based review.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ponytail pass focused only on simplification opportunities.

net: -6 lines possible.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • ab.chatgpt.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "ab.chatgpt.com"

See Network Configuration for more information.

Generated by ✂️ Ponytail Reviewer for #61599 · codex · gpt53codex · 5.44 AIC · ⌖ 3.28 AIC · ⊞ 12.5K
Comment /ponytail to run again

Comments that could not be inline-anchored

pkg/workflow/compiler_experiments.go:69

L69: yagni: separate activation-specific rewrite helper adds API surface for one call site. Inline rewriteDeclaredExperimentNames(..., pickExperimentOutputsPrefix) at use site.

pkg/workflow/compiler_yaml_step_lifecycle.go:179

L179: shrink: temp infoModel variable only feeds one Fprintf. Call RewriteActivationOutputsToLocalStepOutputs(data.Model, data.Experiments) inline in the formatter call.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The activation info step reads experiment outputs before selection occurs, and the regex can corrupt unrelated expression text.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Fixes experiment-based model references when compiling GitHub Actions workflows.

Changes:

  • Adds scoped experiment-reference rewrite helpers.
  • Applies rewrites to model environment variables and activation metadata.
  • Adds unit and compilation regression tests.
File summaries
File Description
pkg/workflow/compiler_experiments.go Adds reference-rewriting helpers.
pkg/workflow/compiler_orchestrator_workflow.go Rewrites configured models after experiment extraction.
pkg/workflow/compiler_yaml_step_lifecycle.go Adapts model references for activation metadata.
pkg/workflow/compiler_experiments_test.go Tests rewrite behavior and name collisions.
pkg/workflow/engine_config_test.go Tests compiled Claude model expressions.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 2
  • Review effort level: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +156 to +160
// data.Model may reference an experiment variant as needs.activation.outputs.<name>
// (rewritten from engine.model: ${{ experiments.<name> }} for jobs downstream of
// activation). This step runs inside the activation job itself, so it must read the
// pick-experiment step's output directly rather than via the needs context.
infoModel := RewriteActivationOutputsToLocalStepOutputs(data.Model, data.Experiments)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b6eda7a: experiment selection steps are now emitted right after the setup steps (addActivationExperimentSteps in newActivationJobBuildContext), so pick-experiment runs before generate_aw_info and GH_AW_INFO_MODEL / the activation model output resolve to the selected variant. Job outputs for experiment names are still registered after the engine outputs so a declared name keeps precedence. Added an ordering assertion in TestCompileClaudeWithExperimentsModel.

Comment on lines +781 to +786
// experimentsFieldReferenceRegex matches `experiments.<name>` tokens (simple identifier)
// appearing anywhere inside a raw configuration string, such as an `engine.model` value.
// Unlike experimentNameRegex/experimentComparisonRegex in expression_extraction.go, this
// pattern is not anchored, so it can rewrite the reference wherever it appears inside a
// larger ${{ ... }} expression (e.g. "${{ experiments.model }}").
var experimentsFieldReferenceRegex = regexp.MustCompile(`\bexperiments\.([a-zA-Z_][a-zA-Z0-9_]*)\b`)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b6eda7a: rewriteDeclaredExperimentNames now only scans the bodies of ${{ ... }} expressions, skips single-quoted string literals (including doubled-quote escapes), and ignores matches adjacent to a .. So ${{ format('experiments.model-{0}', inputs.suffix) }} and ${{ fromJSON(inputs.config).experiments.model }} are left untouched, while ${{ experiments.model || 'sonnet' }} still rewrites. Covered by TestRewriteExperimentsReferenceOnlyRewritesExpressionTokens.

@github-actions

Copy link
Copy Markdown
Contributor
🏗️ ADR Required — draft added for PR #61599

I enforced the design-decision gate for this PR because it adds 228 new lines in business-logic directories (pkg/), which exceeds the default 100-line threshold.

Evidence used

  • adr-prefetch-summary.json: has_implementation_label=false, default_business_additions=228, requires_adr_by_default_volume=true
  • PR title/body: this change fixes how engine.model / top-level model: values using ${{ experiments.model }} are compiled into job-scoped workflow expressions
  • Diff files: compiler rewrite helpers, orchestrator wiring, activation-job rendering, and regression tests in pkg/workflow/*

Gate result

No existing ADR covering this decision was found in the PR body or on the branch, so I added a draft ADR:

  • docs/adr/61599-rewrite-experiment-model-references.md

Next action

Please review and refine the draft ADR so the architectural decision and trade-offs are explicit before merge.

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • github.com
  • proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"
    - "proxy.golang.org"

See Network Configuration for more information.

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · pi · gpt54 · 16.9 AIC · ⊞ 9.9K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the experiments-model rewrite change (refactor/cleanup on gh-aw compiler internals). The logic looks correct and well-tested: RewriteExperimentsReferenceForDownstreamJobs is applied to workflowData.Model right after experiments are extracted, and RewriteActivationOutputsToLocalStepOutputs correctly converts it back for the activation job's own info step, since a job can't reference its own outputs via needs. Word-boundary anchoring in the regexes correctly avoids the model vs model_variant prefix-collision case, which is covered by a dedicated test. Compile-time model validation (validateModelAliasMap, warnCodexCopilotModelCompatibility) runs before the rewrite and already skips values containing ${{, so no regression there.

One non-blocking cleanup item left as an inline comment: RewriteExperimentsReferenceForActivationJob is new, exported, and tested, but has no production caller — the activation job's info step actually goes through the downstream-rewrite-then-local-conversion path instead. Worth removing or wiring in to avoid dead public API surface.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 91.6 AIC · ⌖ 13.8 AIC · ⊞ 8.6K

Comment thread pkg/workflow/compiler_experiments.go Outdated
// info" step), where the experiment variant was just selected by the earlier pick-experiment
// step in the same job. A job cannot reference its own outputs via the `needs` context, so
// `needs.activation.outputs.*` is not valid here.
func RewriteExperimentsReferenceForActivationJob(s string, experiments map[string][]string) string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RewriteExperimentsReferenceForActivationJob is exported and covered by TestRewriteExperimentsReferenceForActivationJob, but it has no production caller anywhere in the codebase — the activation job's own info step actually goes through RewriteExperimentsReferenceForDownstreamJobs (in extractAdditionalConfigurations) followed by RewriteActivationOutputsToLocalStepOutputs (in generateCreateAwInfo). If this helper isn't meant to be used directly elsewhere, consider removing it (and its test) to avoid dead public API surface, or wire it in where it was originally intended so the two forward/backward-conversion paths aren't duplicated logic. @copilot please address this.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 220.2 AIC · ⌖ 16.4 AIC · ⊞ 10.6K
Comment /matt to run again

Comment thread pkg/workflow/compiler_experiments.go Outdated
// info" step), where the experiment variant was just selected by the earlier pick-experiment
// step in the same job. A job cannot reference its own outputs via the `needs` context, so
// `needs.activation.outputs.*` is not valid here.
func RewriteExperimentsReferenceForActivationJob(s string, experiments map[string][]string) string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] RewriteExperimentsReferenceForActivationJob is exported but never called from production code — only RewriteExperimentsReferenceForDownstreamJobs (in compiler_orchestrator_workflow.go) and RewriteActivationOutputsToLocalStepOutputs (in compiler_yaml_step_lifecycle.go) are wired up.

💡 Why this matters

The activation job path is instead handled by rewriting downstream-form (needs.activation.outputs.*) back to step-local form via RewriteActivationOutputsToLocalStepOutputs, which works because workflowData.Model is already rewritten to downstream form earlier in extractAdditionalConfigurations. That leaves RewriteExperimentsReferenceForActivationJob as dead code with no callers outside its own unit test — it adds public API surface and a maintenance burden without being exercised by the actual compile path.

Consider either removing this function (and its dedicated test) if RewriteActivationOutputsToLocalStepOutputs fully covers the activation-job case, or documenting why both entry points need to exist (e.g. a future direct-experiments-to-activation-job path that does not go through the downstream rewrite first).

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed RewriteExperimentsReferenceForActivationJob (and its test) in b6eda7a — it had no production caller; the activation-job path goes through RewriteActivationOutputsToLocalStepOutputs.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot PR sous-chef triage: there are unresolved review threads to address before a maintainer can investigate quickly.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 24.6 AIC · ⊞ 9.6K · ◷
Comment /souschef to run again

Copilot AI and others added 2 commits September 17, 2026 14:22
…okens

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot PR sous-chef triage: maintainer follow-through is still blocked by one unresolved review thread.

Generated by PR Sous Chef run: https://github.com/github/gh-aw/actions/runs/35239680233

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 28.8 AIC · ⊞ 9.6K · ◷
Comment /souschef to run again

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot Please move this PR forward.

Open review threads (newest first):

Please address the feedback, refresh the branch if needed, rerun any affected checks, and then continue with the pr-finisher skill.

Run: https://github.com/github/gh-aw/actions/runs/35240793612

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 24.9 AIC · ⊞ 9.3K · ◷
Comment /souschef to run again

@pelikhan
pelikhan merged commit 7f516a8 into main Sep 17, 2026
1 of 2 checks passed
Copilot stopped work on behalf of gh-aw-bot due to an error September 17, 2026 15:50
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.89.17

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Claude engine: ${{ experiments.model }} not rewritten — workflow fails to parse with 'Unrecognized named-value: experiments'

4 participants