Perf: avoid repeated orchestration-history scans for tracing

[berndverst]

Summary

OnRunOrchestratorAsync in src/Worker/Grpc/GrpcDurableTaskWorker.Processor.cs scanned orchestration history to correlate tracing events unconditionally, and rescanned the full past-events list for every qualifying new event to find the originating TaskScheduled/SubOrchestrationInstanceCreated event — O(new events × past events), even when no tracing listener was registered.

This is a precise, internal-only performance fix:

• Added TraceHelper.HasListeners, a cheap check backed by ActivitySource.HasListeners(), and gated all tracing-correlation work in OnRunOrchestratorAsync behind it, so nothing is scanned when no listener is registered.
• Replaced the LINQ Concat-based scan for the ExecutionStarted event with a single-pass local function.
• Added TraceHistoryEventLookup (src/Shared/Grpc/Tracing/TraceHistoryEventLookup.cs), which builds one dictionary index per work item (lazily, at most once per event type) for O(1) repeated lookups instead of rescanning PastEvents per new event, while preserving the exact original first-wins (SubOrchestrationInstanceCreated) and last-wins (TaskScheduled) semantics for duplicate event IDs.

No public API changes. Trace output, error handling, and orchestration replay determinism are unaffected — this only changes how existing history is looked up, not what is looked up or emitted.

Tests

• test/Worker/Grpc.Tests/TraceHistoryEventLookupTests.cs (new): direct unit tests for TraceHistoryEventLookup covering duplicate-event-ID first/last-wins semantics, no-match, event-type filtering, and empty input.
• test/Worker/Grpc.Tests/GrpcDurableTaskWorkerTests.cs (added):
  • A regression test asserting the no-listener fast path completes work items normally without performing trace-event lookups.
  • An end-to-end test with duplicate event IDs in history that registers a real ActivityListener and asserts the resulting activities reflect the original first/last-wins correlation semantics.
• Full Worker.Grpc.Tests suite passes (145/145).
• Worker.Grpc builds cleanly across all target frameworks (netstandard2.0;net6.0;net8.0;net10.0).

Fixes #769

[Copilot]

Pull request overview

Improves the gRPC worker’s tracing correlation performance by avoiding unnecessary orchestration-history scans when tracing is disabled and by indexing past history events for O(1) correlation lookups when tracing is enabled. This aligns with the SDK’s goal of keeping worker-side orchestration processing efficient, especially for long-running instances with large histories.

Changes:

• Added a cheap listener check (TraceHelper.HasListeners) and gated trace-correlation work in OnRunOrchestratorAsync behind it.
• Replaced repeated past-history scans with a per-work-item indexed lookup (TraceHistoryEventLookup) that preserves prior first/last-wins semantics for duplicate event IDs.
• Added unit and worker-level tests covering listener/no-listener behavior and duplicate-ID correlation semantics.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/Worker/Grpc.Tests/TraceHistoryEventLookupTests.cs	Adds focused unit tests for the new indexed lookup, including duplicate-ID first/last-wins semantics and filtering by event type.
test/Worker/Grpc.Tests/GrpcDurableTaskWorkerTests.cs	Adds end-to-end worker tests for the no-listener fast path and for correlation semantics under a real ActivityListener.
src/Worker/Grpc/GrpcDurableTaskWorker.Processor.cs	Gates tracing correlation behind HasListeners, replaces LINQ concat scan with a single-pass helper, and uses the indexed lookup for repeated correlation.
src/Shared/Grpc/Tracing/TraceHistoryEventLookup.cs	Introduces a lazily-built, cached dictionary index for past events to avoid O(N×M) repeated scans.
src/Shared/Grpc/Tracing/TraceHelper.cs	Exposes HasListeners backed by ActivitySource.HasListeners() to allow callers to skip trace-correlation work when tracing is disabled.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

[berndverst]

Quantitative performance impact

I compared live head d223fc2 with its merge base in a focused Release/.NET 10 harness on Windows x64 (AMD EPYC 7763). It used the actual TraceHistoryEventLookup/TraceHelper.HasListeners code and the exact old lookup algorithm. Activity creation, the rest of worker execution, and gRPC were excluded, so these are correlation-stage measurements rather than fleet-wide orchestration averages.

Let H be past-event count and N_t/N_s qualifying task/sub-orchestration completion counts.

Path	Old past-event inspections	New work
No tracing listener	execution-start search + up to N_t*H + sum(sub match positions)	one HasListeners() check; zero history enumeration
Listener, task completions	N_t*H	one H-event task index + N_t dictionary lookups
Listener, sub-orchestration completions	sum(sub match positions)	one H-event sub index + N_s dictionary lookups

Representative, best, and adverse cases

“Representative” below means a selected ordinary shape, not production telemetry.

Case	Listener / shape	Correlation latency, old -> new	Allocation/work item	Interpretation
Average-style representative	Off; H=100, 5 task completions	4.40 us -> ~6 ns	1,056 -> 0 B	The history work is eliminated; the new value is near harness overhead, so the ratio is not meaningful
Listener-on ordinary task case	H=100, 5 task completions	4.38 -> 2.00 us (-54.3%)	1,056 -> 4,704 B (+3,648 B)	Faster CPU path, higher temporary allocation
Listener-on ordinary mixed case	H=100, 3 task + 3 early sub matches	2.55 -> 3.43 us (~+34.5%)	1,024 -> 4,280 B (+3,256 B)	Small/early-match histories can regress
Best measured large replay	Off; H=10,000, 1,000 task completions	79.8 ms -> near harness overhead	160,256 -> 0 B	Removes the entire quadratic correlation scan
Listener-on large fan-out	H=10,000, 1,000 task completions	80.46 ms -> 333.5 us (-99.585%, ~241x)	160,256 -> 451,592 B (+291,336 B)	Large CPU/latency gain traded for a temporary index
Listener-on mixed fan-out	H=10,000, 500 task + 500 sub completions	36.98 ms -> 490.7 us (-98.673%, ~75x)	128,256 -> 431,438 B (+303,182 B)	Same tradeoff with two lazy indexes
Constructed adverse case	Listener on; one sub match at position 2 of H=10,000	166 ns -> 95.7 us (~576x as long)	352 -> 296 B	Old FirstOrDefault stops at event 2; new code builds the full lazy index

The no-listener ~6 ns figures are close to benchmark-harness cost and should be read as “the scan was removed,” not as a credible millions-of-operations throughput claim.

Memory and GC

The lazy indexes reference existing events; they do not copy protobuf messages. On this x64 runtime, estimated incremental dictionary storage while live is approximately 28-56 B per unique indexed ID, including capacity slack/array headers but excluding the existing event objects.

• 50 task entries: about 2.7 KiB retained, 4.7 KiB cumulatively allocated
• 5,000 task entries: about 236 KiB retained, 452 KiB allocated
• Two 3,333-entry indexes: about 227 KiB retained, 431 KiB allocated

Thus:

• No listener: both scan CPU and correlation allocations fall to zero.
• Listener + repeated lookups: CPU falls from O(N*H) to O(H+N), but temporary memory becomes O(unique history IDs).
• Listener + one early sub lookup: CPU can regress from O(1) expected early exit to O(H) index construction.
• A listener that samples every activity out still makes HasListeners() true and pays indexing costs.

Whole-worker throughput interpretation

Correlation-stage speedup does not translate directly to orchestration throughput. Using Amdahl's law:

• If the old correlation work is 10% of turn CPU and the new path removes it, modeled turn throughput improves by about 11.1%.
• If a large fan-out replay spends 50% of turn CPU in old correlation scans and that work becomes negligible, the modeled ceiling is about 2x turn throughput.

Those are sensitivity examples. Actual gains require telemetry for listener prevalence and the distributions of H, qualifying new events, event mix, and match positions.