[release/11.0] Backport Helix Job Monitor improvements

[mmitche]

Backports the Helix Job Monitor changes that landed on main after release/11.0 branched.

Included changes:

• Fix swagger generator & regenerate Helix client #17372 — Fix Swagger generator and regenerate the Helix client (required for property-filtered job queries)
• Job Monitor: Search associated jobs more efficiently #17373 — Search associated jobs more efficiently
• Rewrite the Helix Job Monitor for scalable result processing #17331 — Rewrite the Helix Job Monitor for scalable result processing
• Shorten data-driven test result names #17303 — Shorten data-driven test result names
• Restore stable test storage for Helix results #17429 — Restore stable test storage for Helix results
• Include version in Job Monitor user agent #17430 — Include the version in the Job Monitor user agent

Validation:

• Build.cmd -test -projects src\Microsoft.DotNet.Helix\Sdk.Tests\Microsoft.DotNet.Helix.Sdk.Tests\Microsoft.DotNet.Helix.Sdk.Tests.csproj

[Copilot]

Pull request overview

Backports a set of post-branch main improvements to the Helix Job Monitor onto release/11.0, including updated Swagger/Handlebars code generation, a regenerated Helix C# client with property-based job filtering, and a substantial Job Monitor rewrite focused on scalable, bounded, streaming result processing with richer metrics and improved retry/replay semantics.

Changes:

• Fix Swagger generator compatibility with newer Handlebars.NET and regenerate the Helix client (including property-filtered job queries and updated models).
• Replace the legacy fire-and-forget upload queue with a bounded, multi-stage upload pipeline (download → publish → finalize) plus rate-limit coordination and metrics.
• Update/expand unit and scenario tests plus documentation/design notes to reflect the new behavior and configuration defaults (e.g., upload parallelism, .xml.txt discovery, versioned user-agent).

Reviewed changes

Copilot reviewed 62 out of 73 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/Microsoft.DotNet.SwaggerGenerator/Microsoft.DotNet.SwaggerGenerator.CodeGenerator/Modeler/ServiceClientModelFactory.cs	Adds null-safe handling around OpenAPI extensions/content during modeling.
src/Microsoft.DotNet.SwaggerGenerator/Microsoft.DotNet.SwaggerGenerator.CodeGenerator/Languages/csharp/ServiceClient.hb	Updates generated exception serialization members with explicit [Obsolete] message; trims whitespace.
src/Microsoft.DotNet.SwaggerGenerator/Microsoft.DotNet.SwaggerGenerator.CodeGenerator/HelperFactory.cs	Reworks Handlebars helper binding to reflection-based invocation compatible with newer Handlebars.NET APIs.
src/Microsoft.DotNet.Helix/Sdk/Readme.md	Links Job Monitor design docs and summarizes new streaming/bounded processing behavior.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/TestNameFormatterTests.cs	Adds coverage for data-driven sub-result display-name formatting.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/ScenarioHelpers/ScenarioHelpers.cs	Extends scenario helpers for attempts/phases and richer Helix job metadata in tests.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests.csproj	Removes reference to the standalone AzureDevOpsTestPublisher project.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/LocalTestResultsReaderTests.cs	Adds .txt suffix recognition tests and updates xUnit file-name expectations.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/HelixServiceTests.cs	Updates Helix service tests for property-filtered job queries and per-work-item downloads/resubmits.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/Fakes/FakeHelixService.cs	Updates fake Helix service surface to per-work-item download and attempt-aware resubmissions; adds call counting.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/Fakes/FakeAzureDevOpsService.cs	Updates fake AzDO service to per-work-item upload summaries and tracks max concurrent uploads.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/AzureDevOpsServiceTests.cs	Adds tests for versioned user-agent, retry/rate-limit behavior, and request-kind metrics.
src/Microsoft.DotNet.Helix/Sdk.Tests/Microsoft.DotNet.Helix.Sdk.Tests/AzureDevOpsResultPublisherTests.cs	Adds extensive batching/streaming/identity/attachment behavior coverage for the new publisher path.
src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs	Adjusts queue health logging for new generated QueueStatsSummary shape.
src/Microsoft.DotNet.Helix/JobMonitor/TestResultUploadQueue.cs	Removes the old upload queue implementation (replaced by pipeline).
src/Microsoft.DotNet.Helix/JobMonitor/TestResultUploadPipeline.cs	Introduces the bounded, staged upload pipeline and drain/cancel behavior.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/TestResultUploadSummary.cs	Defines a shared upload summary record used by pipeline/state.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/TestNameFormatter.cs	Extends formatter to preserve display names for data-driven sub-results.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/ResultAggregator.cs	Adds aggregation logic for single/rerun/data-driven results and FQN-based identity grouping.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/Model/TestResultAttachmentMode.cs	Moves attachment mode enum into JobMonitor-owned model namespace.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/Model/TestResultAttachment.cs	Defines attachment model used during parse/publish.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/Model/TestResult.cs	Defines parsed test result model with stable fully-qualified identity.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/Model/TerminalError.cs	Adds terminal error type for non-retryable/reporting-stopping failures.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/Model/PackedTestReport.cs	Defines a packed report container for transport/publishing.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/Model/AzureDevOpsReportingParameters.cs	Defines AzDO publishing parameters model.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/Model/AzureDevOpsReportingError.cs	Defines non-terminal reporting error type.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/LocalTestResultsReader.cs	Adds streaming XML parsing for xUnit/JUnit/TRX with .txt suffix handling and attachment selection.
src/Microsoft.DotNet.Helix/JobMonitor/TestResults/IAzureDevOpsResultTransport.cs	Introduces internal transport abstraction for AzDO result/attachment publishing.
src/Microsoft.DotNet.Helix/JobMonitor/StatusReporter.cs	Refactors status logging to consume poll snapshots and pipeline counters (no extra service calls).
src/Microsoft.DotNet.Helix/JobMonitor/Services/HelixService.cs	Switches job discovery to server-side property filtering + per-work-item downloads; adds metrics hooks and attempt stamping.
src/Microsoft.DotNet.Helix/JobMonitor/Services/AzureDevOpsService.cs	Adds versioned user-agent, unified request pipeline with throttling gate, retries, and metrics; refactors upload path to per-work-item.
src/Microsoft.DotNet.Helix/JobMonitor/Services/AzureDevOpsRateLimitGate.cs	Adds shared gate to coordinate rate-limit backoff across concurrent workers.
src/Microsoft.DotNet.Helix/JobMonitor/Parallelism/ParallelAsync.cs	Adds helper for bounded parallel snapshot reads into a dictionary.
src/Microsoft.DotNet.Helix/JobMonitor/Parallelism/ActionQueue.cs	Adds reusable channel-based bounded/unbounded worker queues with counters, cancel, and drain.
src/Microsoft.DotNet.Helix/JobMonitor/MonitorState.cs	Updates logical-stream keying/outcome ordering for attempt-aware retries and incremental upload failures.
src/Microsoft.DotNet.Helix/JobMonitor/Models/HelixJobInfo.cs	Adds attempt/phase/logical-job metadata and resubmission diagnostics properties.
src/Microsoft.DotNet.Helix/JobMonitor/Microsoft.DotNet.Helix.JobMonitor.csproj	Enables implicit usings/nullable annotations and removes old publisher project reference.
src/Microsoft.DotNet.Helix/JobMonitor/JobMonitorOptions.cs	Adds --job-attempt + defaults and increases default upload parallelism to 48.
src/Microsoft.DotNet.Helix/JobMonitor/JobMonitorMetrics.cs	Adds shared atomic metrics for requests, retries, throttling, payload, and pipeline stage timing.
src/Microsoft.DotNet.Helix/JobMonitor/Interfaces/IHelixService.cs	Updates interface to require server-side build filtering and per-work-item downloads + monitor job attempt.
src/Microsoft.DotNet.Helix/JobMonitor/Interfaces/IAzureDevOpsService.cs	Updates interface to upload one work item at a time (returning a single summary).
src/Microsoft.DotNet.Helix/JobMonitor/Design/README.md	Adds design index for the new Job Monitor architecture.
src/Microsoft.DotNet.Helix/JobMonitor/Design/Components/UploadPipeline.md	Documents the bounded pipeline stages, defaults, and durability boundary.
src/Microsoft.DotNet.Helix/JobMonitor/Design/Components/TestResults.md	Documents streaming parsing, aggregation, batching, and .txt artifact suffix recognition.
src/Microsoft.DotNet.Helix/JobMonitor/Design/Components/StateAndStatus.md	Documents state ownership, snapshots, and bounded verbose logging.
src/Microsoft.DotNet.Helix/JobMonitor/Design/Components/Shutdown.md	Documents drain vs cancellation semantics and crash recovery assumptions.
src/Microsoft.DotNet.Helix/JobMonitor/Design/Components/Polling.md	Documents poll snapshot reuse and attempt-aware reconciliation approach.
src/Microsoft.DotNet.Helix/JobMonitor/Design/Components/PerformanceMetrics.md	Documents aggregated metrics and how to interpret them.
src/Microsoft.DotNet.Helix/JobMonitor/Design/Architecture.md	Documents ownership boundaries, dataflow, and shared parallelism primitives.
src/Microsoft.DotNet.Helix/Client/CSharp/RequestUriBuilderExtensions.cs	Adds query builder helper for Properties[key]=value filters in generated client.
src/Microsoft.DotNet.Helix/Client/CSharp/QueueStats.cs	Removes handwritten model replaced by regenerated API model(s).
src/Microsoft.DotNet.Helix/Client/CSharp/JobStatus.cs	Removes handwritten shim replaced/obsolete in regenerated API surface.
src/Microsoft.DotNet.Helix/Client/CSharp/JobCreationResult.cs	Removes handwritten shim now covered by regenerated client code.
src/Microsoft.DotNet.Helix/Client/CSharp/HelixApi.cs	Updates client wiring/usings for regenerated surface area.
src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Models/QueueStatsSummary.cs	Adds regenerated queue stats summary model.
src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Models/QueueInfo.cs	Adds new regenerated queue properties (protected mode metadata).
src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Models/JobStatus.cs	Removes regenerated type no longer present in the API doc.
src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Models/JobInfo.cs	Removes regenerated type no longer present in the API doc.
src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Models/JobCreationResult.cs	Updates regenerated job creation result model (queue stats + validation).
src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs	Adds properties parameter support and query emission for job listing.
src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/HelixApi.cs	Removes telemetry surface, adjusts pipeline creation, and updates obsolete attributes in generated exceptions.
src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Aggregate.cs	Adds properties parameter support for aggregate endpoints.
src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/Microsoft.DotNet.Helix.AzureDevOpsTestPublisher.csproj	Removes standalone publisher project (now folded into Job Monitor).
src/Microsoft.DotNet.Helix/AzureDevOpsTestPublisher/LocalTestResultsReader.cs	Removes old non-streaming reader implementation (replaced by Job Monitor version).
eng/common/core-templates/job/helix-job-monitor.yml	Adds parameters and forwards job attempt + upload parallelism to the monitor tool.
Arcade.slnx	Removes deleted AzureDevOpsTestPublisher project from the solution.
.gitignore	Ensures JobMonitor/TestResults/** is not accidentally ignored by broad test-results patterns.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.