Skip to content

[PERF] Fix Helix monitor credentials in the runtime performance pipeline - #133633

Merged
mmitche merged 1 commit into
dotnet:mainfrom
LoopedBard3:loopedbard3-helix-wait-regression
Sep 10, 2026
Merged

[PERF] Fix Helix monitor credentials in the runtime performance pipeline#133633
mmitche merged 1 commit into
dotnet:mainfrom
LoopedBard3:loopedbard3-helix-wait-regression

Conversation

@LoopedBard3

@LoopedBard3 LoopedBard3 commented Sep 10, 2026

Copy link
Copy Markdown
Member

Summary

Fix the regression introduced by #132807 (Enable Helix job monitor for runtime perf) by making DotNet-HelixApi-Access available at the Build stage scope in eng/pipelines/performance/perf.yml for internal, non-PR runs.

The standalone monitor now inherits the credential needed to discover private Helix jobs. The change is limited to the variable-group import; it leaves the SDK, asynchronous submission, and existing monitor options unchanged.

Regression

The original PR intentionally made Send job to Helix return after submission and delegated completion tracking to Monitor Helix Jobs. However, the Helix credential group was imported only inside the individual build/submission jobs. The new sibling monitor did not inherit it, leaving its $(HelixApiAccessToken) reference unresolved.

Helix returns an empty successful discovery response without valid authentication. Combined with allowNoHelixJobs: true, this allowed the pipeline to succeed while submitted performance work was still queued.

The transition is visible between 20260908.2, before the original PR and 20260908.3, at its merge commit. Both had the same runtime Arcade SDK pin, so the later SDK update was not the trigger.

Before / after

Before: Monitor log from 20260909.9 showed an all-zero Helix status throughout the run and finished with:

Jobs:       0 submitted / 0 resubmitted / 0 processed
Work items: 0 submitted / 0 resubmitted / 0 failed

After: Internal sanity run 20260910.11 / build 3072330, running this commit with onlySanityCheck=true, now shows actual Helix work items and their outcomes in the monitor rather than the all-zero state. For example, the monitor reports arm64.micro.net11.0.Partition0 from job 40418dd2-e32c-4474-bd8f-e9bf510b7708 and x86.micro.net11.0.Partition0 from job 4e7b9e80-2a9a-4da8-81a7-731ad8752366, including their finished state and exit code.

This confirms private-job discovery and outcome reporting. The run is still in progress at the time of opening this PR; full end-to-end completion is not yet claimed.

The sampled ARM64/x86 work-item failures are NETSDK1045 benchmark-generation failures also present in main build 3072178. Two iOS setup jobs encountered a CertHelper NuGet restore failure also present in main build 3072012. These are separate from the credential-scope fix; the repaired monitor is now surfacing real work-item failures that were previously missed.

Validation

  • Parsed the YAML and checked that the only structural difference is the guarded stage-level variable-group import.
  • git diff --check.
  • Queued internal definition 702 on commit e6f8fbe40248c99b6ae62ebeef27ff40552d5bea with onlySanityCheck=true; monitoring remains in progress as described above.

Note

This PR description and fix were generated with GitHub Copilot.

@LoopedBard3
Provide Helix credentials to the performance job monitor
e6f8fbe 
Import the Helix access variable group at Build-stage scope for internal non-PR runs so the standalone monitor can discover submitted jobs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings September 10, 2026 20:00
@LoopedBard3 LoopedBard3 mentioned this pull request Sep 10, 2026
Closed
@azure-pipelines

azure-pipelines Bot commented Sep 10, 2026

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 6 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@LoopedBard3

LoopedBard3 commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

FYI @mmitche and @akoeplinger. Also, if this is not the correct fix let me know and we can fix it the correct way, this was just what copilot found and seemed to work internally.

@LoopedBard3 LoopedBard3 self-assigned this Sep 10, 2026
@LoopedBard3 LoopedBard3 added the perf-pipeline Issues with dotnet-runtime-perf, or runtime-wasm-perf pipelines label Sep 10, 2026
Copilot AI reviewed Sep 10, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is narrowly scoped to stage-level variable-group availability and follows established pipeline patterns for conditional variable group imports.

Pull request overview

This PR adjusts the runtime performance pipeline to ensure the Helix Job Monitor job can access the DotNet-HelixApi-Access variable group by importing it at the Build stage scope for internal, non-PR runs, so the monitor can authenticate when discovering and tracking Helix jobs.

Changes:

  • Add a variables section at the Build stage level that conditionally imports the DotNet-HelixApi-Access variable group when System.TeamProject == internal and Build.Reason != PullRequest.
File summaries
File Description
eng/pipelines/performance/perf.yml Imports DotNet-HelixApi-Access at Build stage scope (guarded to internal non-PR runs) so the standalone Helix monitor can resolve $(HelixApiAccessToken).
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

@LoopedBard3
LoopedBard3 requested a review from mangod9 September 10, 2026 20:09
@LoopedBard3 LoopedBard3 changed the title Fix Helix monitor credentials in the runtime performance pipeline [PERF] Fix Helix monitor credentials in the runtime performance pipeline Sep 10, 2026
@LoopedBard3
LoopedBard3 enabled auto-merge (squash) September 10, 2026 20:34
@mmitche

mmitche commented Sep 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Thanks, this makes sense given the internal pipeines.

@mmitche
mmitche merged commit dc71a6d into dotnet:main Sep 10, 2026
65 of 79 checks passed
@LoopedBard3
LoopedBard3 deleted the loopedbard3-helix-wait-regression branch September 10, 2026 22:07
This was referenced Sep 11, 2026
Open
Closed
@dotnet-maestro dotnet-maestro Bot mentioned this pull request Sep 11, 2026
Merged
@akoeplinger

akoeplinger commented Sep 11, 2026

Copy link
Copy Markdown
Member

/backport to release/11.0

@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Started backporting to release/11.0 (link to workflow run)

@github-actions github-actions Bot mentioned this pull request Sep 11, 2026
Merged
@akoeplinger akoeplinger mentioned this pull request Sep 11, 2026
Merged
akoeplinger added a commit that referenced this pull request Sep 11, 2026
@akoeplinger
Add DotNet-HelixApi-Access variable group for runtime.yml (#133688)
c5cc451 
Same as #133633

This can go away once dotnet/arcade#17537 lands
akoeplinger pushed a commit that referenced this pull request Sep 11, 2026
@github-actions
[release/11.0] [PERF] Fix Helix monitor credentials in the runtime pe…
d23a49d 
…rformance pipeline (#133681)

Backport of #133633 to release/11.0
This was referenced Sep 11, 2026
Merged
Merged
@dotnet-milestone-bot dotnet-milestone-bot Bot added this to the 12.0-preview1 milestone Sep 14, 2026
@akoeplinger akoeplinger mentioned this pull request Sep 15, 2026
Merged
LoopedBard3 added a commit that referenced this pull request Sep 16, 2026
@LoopedBard3 @akoeplinger
[PERF] Use standalone Helix monitor for slow performance jobs (#133726)
8185d13 
## Summary

Adopt the standalone Helix Job Monitor for `perf-slow`, following
#132807, the stage-scoped credential fix in
#133633, and the internal-PR credential alignment proposed
in #133885.

- Always enable asynchronous submission and include the standalone
monitor job with an explicit parameter block, including on public and
no-selection runs. Existing benchmark selectors are unchanged; no public
benchmark workloads are added.
- Import `DotNet-HelixApi-Access` at stage scope for all
internal-project runs, including PRs, since the sibling monitor does not
inherit submitter job variable groups. Always include
`helixAccessToken`, using a compile-time `iif` expression to select
`$(HelixApiAccessToken)` only for the internal project and an empty
string for anonymous access elsewhere. Public runs, including public
PRs, receive no private group or token.
- Keep a six-hour monitor job cap (`360` minutes; the tool's maximum
wait is `355` minutes). Existing submitter and workitem timeouts remain
unchanged, so this is not a whole-pipeline wall-clock deadline.
- Allow legitimate zero-submission runs and preserve the existing
private/scheduled job selection, sanity flag, triggers, and schedules.

Only `eng/pipelines/performance/perf-slow.yml` changes.

## Validation

Internal pipeline **1012**, [sanity run
3072457](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072457),
ran with `onlySanityCheck=true`, `runScheduledJobs=true`, and
`runPrivateJobs=true`.

- Tested runtime commit: `12fb88f1065a13388d7488fd28e65e18006a08f2`.
- Pinned performance resource:
`c735123e8b45db1c8d0e30207bfe24a5fca2621c`.
- All four build jobs and five Azure submitter jobs succeeded. Sends
completed in 29-45 seconds, before their Helix workloads finished.
- At 22:23:48 UTC, monitor log **239** recorded nine Azure jobs
completed, none running or waiting, while one Helix job/workitem was
still running. The monitor was the only remaining Azure job.
- After the final workitem finished at 22:24:06 UTC, the monitor drained
all results and exited with code 1. Its final summary was **5 jobs
submitted, 0 resubmitted, 5 processed; 5 workitems, 5 failed; 5 results
uploaded**. The pipeline correctly finished **failed**, not false-green.

This demonstrates private job discovery, asynchronous agent release,
completion gating, and failure propagation. **It is not a fully passing
benchmark run.**

### Failure caveats

Four Linux workitems failed with `NETSDK1045`: generated benchmark
projects target .NET 12.0 while the SDK reports support through .NET
11.0. The same error was verified in [main's slow-performance run
3072225](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072225),
including Linux Helix job `59558d46-2044-4bbe-9b59-cc369574171a`,
`arm64.micro.net11.0.Partition0`, console line 848.

The Windows workitem failed differently: `WinError 225` blocked `dotnet
--info` for the downloaded SDK as a virus or potentially unwanted
software. This is **unclassified**, not a confirmed false positive or
confirmed pre-existing failure. Evidence: Helix job
`071489c5-3f06-4671-b957-0a25abcd5d65`,
`arm64.micro.net11.0.Partition0`, console line 436. Build Analysis
supplied no known-issue match.

### Final follow-up coverage

Subsequent revisions decouple private credentials from monitor
eligibility. Commit `1b1a9defdfb3d4a1e1e0b47ba5edaeeb0794ce0e` now
includes the flag, job, and all three parameters unconditionally. These
changes were **not included in the live run** at
`12fb88f1065a13388d7488fd28e65e18006a08f2`.

Commit `7b7d4d1bb9319a3f377126a1139e8f60b095db11` also aligns both
credential guards with the internal-project-only predicate in
#133885 (reference head
`0250fbf77bfe09a37256a550a2a3de35414444e9`, open when inspected). It
enables credentials for internal PRs while leaving public PRs anonymous.
This revision has no new live-run coverage.

The final version passed local unique-key YAML parsing and 192
project/reason/private/scheduled/sanity combinations, including 68
public or no-selection cases with a monitor but no benchmark
submissions. Checks cover unconditional monitor inclusion, credential
import and token selection for internal runs including PRs, an explicit
empty token for all other projects, the 360/355-minute limits, preserved
workload selection/triggers/schedules, the monitor-template contract,
and `git diff --check`.

An Azure preview of the earlier unconditional-monitor revision was
attempted but denied because `EditBuild` permission is required. No
permissions were changed or bypassed. The local expression checks are
not an Azure template compiler or a live internal-PR/public-access test,
and no new manual CI run or local product build was performed for this
pipeline-only revision.

> [!NOTE]
> This change and PR description were prepared with GitHub Copilot.

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com>
Copilot-Session: 5c9e5ce3-0d1b-440d-9c69-78918c5f6dc5
jtschuster pushed a commit to jtschuster/runtime that referenced this pull request Sep 18, 2026
@LoopedBard3
@akoeplinger @jtschuster
[PERF] Use standalone Helix monitor for slow performance jobs (dotnet…
8b3edcf 
…#133726)

## Summary

Adopt the standalone Helix Job Monitor for `perf-slow`, following
dotnet#132807, the stage-scoped credential fix in
dotnet#133633, and the internal-PR credential alignment proposed
in dotnet#133885.

- Always enable asynchronous submission and include the standalone
monitor job with an explicit parameter block, including on public and
no-selection runs. Existing benchmark selectors are unchanged; no public
benchmark workloads are added.
- Import `DotNet-HelixApi-Access` at stage scope for all
internal-project runs, including PRs, since the sibling monitor does not
inherit submitter job variable groups. Always include
`helixAccessToken`, using a compile-time `iif` expression to select
`$(HelixApiAccessToken)` only for the internal project and an empty
string for anonymous access elsewhere. Public runs, including public
PRs, receive no private group or token.
- Keep a six-hour monitor job cap (`360` minutes; the tool's maximum
wait is `355` minutes). Existing submitter and workitem timeouts remain
unchanged, so this is not a whole-pipeline wall-clock deadline.
- Allow legitimate zero-submission runs and preserve the existing
private/scheduled job selection, sanity flag, triggers, and schedules.

Only `eng/pipelines/performance/perf-slow.yml` changes.

## Validation

Internal pipeline **1012**, [sanity run
3072457](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072457),
ran with `onlySanityCheck=true`, `runScheduledJobs=true`, and
`runPrivateJobs=true`.

- Tested runtime commit: `12fb88f1065a13388d7488fd28e65e18006a08f2`.
- Pinned performance resource:
`c735123e8b45db1c8d0e30207bfe24a5fca2621c`.
- All four build jobs and five Azure submitter jobs succeeded. Sends
completed in 29-45 seconds, before their Helix workloads finished.
- At 22:23:48 UTC, monitor log **239** recorded nine Azure jobs
completed, none running or waiting, while one Helix job/workitem was
still running. The monitor was the only remaining Azure job.
- After the final workitem finished at 22:24:06 UTC, the monitor drained
all results and exited with code 1. Its final summary was **5 jobs
submitted, 0 resubmitted, 5 processed; 5 workitems, 5 failed; 5 results
uploaded**. The pipeline correctly finished **failed**, not false-green.

This demonstrates private job discovery, asynchronous agent release,
completion gating, and failure propagation. **It is not a fully passing
benchmark run.**

### Failure caveats

Four Linux workitems failed with `NETSDK1045`: generated benchmark
projects target .NET 12.0 while the SDK reports support through .NET
11.0. The same error was verified in [main's slow-performance run
3072225](https://dev.azure.com/dnceng/internal/_build/results?buildId=3072225),
including Linux Helix job `59558d46-2044-4bbe-9b59-cc369574171a`,
`arm64.micro.net11.0.Partition0`, console line 848.

The Windows workitem failed differently: `WinError 225` blocked `dotnet
--info` for the downloaded SDK as a virus or potentially unwanted
software. This is **unclassified**, not a confirmed false positive or
confirmed pre-existing failure. Evidence: Helix job
`071489c5-3f06-4671-b957-0a25abcd5d65`,
`arm64.micro.net11.0.Partition0`, console line 436. Build Analysis
supplied no known-issue match.

### Final follow-up coverage

Subsequent revisions decouple private credentials from monitor
eligibility. Commit `1b1a9defdfb3d4a1e1e0b47ba5edaeeb0794ce0e` now
includes the flag, job, and all three parameters unconditionally. These
changes were **not included in the live run** at
`12fb88f1065a13388d7488fd28e65e18006a08f2`.

Commit `7b7d4d1bb9319a3f377126a1139e8f60b095db11` also aligns both
credential guards with the internal-project-only predicate in
dotnet#133885 (reference head
`0250fbf77bfe09a37256a550a2a3de35414444e9`, open when inspected). It
enables credentials for internal PRs while leaving public PRs anonymous.
This revision has no new live-run coverage.

The final version passed local unique-key YAML parsing and 192
project/reason/private/scheduled/sanity combinations, including 68
public or no-selection cases with a monitor but no benchmark
submissions. Checks cover unconditional monitor inclusion, credential
import and token selection for internal runs including PRs, an explicit
empty token for all other projects, the 360/355-minute limits, preserved
workload selection/triggers/schedules, the monitor-template contract,
and `git diff --check`.

An Azure preview of the earlier unconditional-monitor revision was
attempted but denied because `EditBuild` permission is required. No
permissions were changed or bypassed. The local expression checks are
not an Azure template compiler or a live internal-PR/public-access test,
and no new manual CI run or local product build was performed for this
pipeline-only revision.

> [!NOTE]
> This change and PR description were prepared with GitHub Copilot.

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Alexander Köplinger <alex.koeplinger@outlook.com>
Copilot-Session: 5c9e5ce3-0d1b-440d-9c69-78918c5f6dc5
akoeplinger added a commit that referenced this pull request Sep 21, 2026
@akoeplinger
[release/10.0] Backport Helix job monitor (#133959)
ca3bfb8 
## Description

Backport the Helix job monitor integration from main/release/11.0,
including the groundwork from #129690 and #132150 and the re-enablement
in #131969.

Includes the subsequent fixes for empty stages (#132019), conditional
monitor inclusion (#132882, #132884), parameter forwarding (#133002),
performance monitoring (#132807, #133480), and internal credentials
(#133633, #133688, #133885).

- Use `Microsoft.DotNet.Helix.JobMonitor` version
`10.0.0-beta.26461.103`, matching release/10.0's existing Arcade/VMR
build.
- Preserve the existing SDK, shared Arcade templates, queues, and
release/10.0 job layouts.
- Follow upstream enablement, except scheduled libraries outerloop runs
retain release/10.0's existing warning-only reporting policy.
- Leave SuperPMI's post-Helix processing unchanged. The separate
perf-slow enablement in #133726 is not included.

## Customer Impact

CI infrastructure only; no shipped runtime changes. Moves Helix waiting
and test-result reporting into the standalone monitor for the enabled
pipelines.

## Regression

Not a product regression fix; backports existing CI infrastructure and
its follow-up fixes.

## Testing

- Validated YAML/JSON/XML configuration and preservation of unrelated
settings.
- Checked 43 entry pipelines, 149 forwarding sites, and 516
public/internal, PR/scheduled/manual, and normal/staging combinations,
plus disabled-mode behavior.
- Exercised MSBuild child-property forwarding and the pinned SDK's
waiting/reporter properties, including environment-based opt-in.
- Restored the monitor and verified its CLI compatibility under .NET 10.

No product build or live Azure DevOps pipeline execution was performed
locally.

## Risk

Changes CI scheduling and result reporting, not product behavior. Uses
the monitor package matching the branch's existing toolset and retains
the release/10.0-specific scheduled outerloop failure policy. End-to-end
Azure DevOps validation is still required.

> [!NOTE]
> This PR description and backport were prepared with GitHub Copilot.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: aae9b201-32a8-45eb-ae1a-db80a0ab4156
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@mmitche mmitche mmitche approved these changes
@mangod9 mangod9 mangod9 approved these changes
@DrewScoggins DrewScoggins Awaiting requested review from DrewScoggins

Assignees

@LoopedBard3 LoopedBard3

Labels

area-Infrastructure perf-pipeline Issues with dotnet-runtime-perf, or runtime-wasm-perf pipelines

Projects

None yet

Milestone

12.0-preview1

Development

Successfully merging this pull request may close these issues.

5 participants

@LoopedBard3 @mmitche @akoeplinger @mangod9