Add an EventPipe-backed GC pause duration histogram

[matyaskollert]

Implements the GC-pause histogram requested in #125753 using EventPipe, as an alternative to #133941. No new public managed API or private CoreLib dependency is introduced.

Metric	Value
Meter / instrument	System.Runtime / dotnet.gc.pause.duration
Type / unit	Histogram<double> / s
Tags	gc.heap.generation: gen0, gen1, gen2; gc.pause.type: blocking, background

The shared GC emits a typed GCPause_V1 event for each existing pause-accounting contribution. DiagnosticSource consumes it through public EventListener APIs and records the duration in seconds. Background contributions remain separate, LOH/POH collections are attributed to Gen 2, and the cumulative dotnet.gc.pause.time counter is unchanged.

The histogram remains available but produces no samples when the runtime cannot provide the events or tracing is disabled. This removes the GCPauseReporting.IsSupported() helper, reflection/UnsafeAccessor bridge and capability QCall. Delivery remains best-effort; buffer overflow and session changes can lose records, and no reliable dropped-record count is available.

Why the dispatcher change is needed

EventPipeEventDispatcher.Threads.cs must capture each session's cancellation token before scheduling its task. Otherwise, a delayed old task can read a newer session's token, continue with the old keyword filter and block the replacement task. Adding the histogram's keyword can then produce no measurements. The included regression test deliberately delays task startup and changes the keyword set; it fails without the fix and passes with it.

Validation

• CoreCLR and NativeAOT native/CoreLib builds passed; all 20 focused cases and 468 DiagnosticSource cases passed. Focused cases also passed repeated Checked and Release runs.
• Both the disabled-tracing behavior and dispatcher regression were shown to fail before their fixes. Test hardening covers shared-session reconfiguration, actual condemned generations, and actionable timeout diagnostics.
• Published NativeAOT Workstation/Server apps passed with tracing enabled and disabled, including exact per-generation measurements and listener lifecycle. The in-process NativeAOT regression passed. An older CoreCLR without the event exposed an empty histogram without throwing.
• Non-Windows CI coverage remains to be confirmed. Native GC-to-EE version guards are retained.

Performance

Windows x64 Release GCPerfSim with histogram aggregation enabled (measured revision: febbedf0bde). Five processes per profile/variant, three-second warmup and at least 15 seconds of measured work, with background GC disabled.

Throughput	Main	EventPipe
Workstation, small objects	10.172 GB/s	10.295 GB/s (+1.21%)
Server, request-style	12.739 GB/s	12.831 GB/s (+0.73%)

Both paired 95% throughput intervals include zero (-2.49% to +5.10%, and -0.72% to +2.19%), so these measurements establish neither a throughput improvement nor a regression. All 50,135 contributions matched native counts and duration totals.

Related to #125753.

[azure-pipelines]

Azure Pipelines:
Successfully started running 4 pipeline(s).
12 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @dotnet/area-system-diagnostics-tracing
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @anicka-net, @dotnet/gc
See info in area-owners.md if you want to be subscribed.

[matyaskollert]

I am not yet sure if we require an #ifdef NET_11_0_OR_GREATER equivalent for when it is released for .net12, but I kept it in the implementation for now

[jkotas]

We should think about the end-to-end experience and whether this histogram should be part of the default .NET runtime counter set.

Tools like dotnet-counters are going to display this histogram by default since it is in the default set. The histogram can take a lot of space to display. The default .NET runtime counters take ~40 lines in dotnet-counters. The histogram takes another up to 16 lines (that is up 40% more lines).

Should the histogram be in some kind of opt-in counter set?

[jkotas]

For reference, this is what the default dotnet-counters looks like currently vs. with this change

[kkokosa]

some kind of opt-in counter set

Do we have something like this? Would be good inded.

[matyaskollert]

I do not believe such an opt-in counter set exists yet, but it could be added if filtering using --counters is not sufficient

[lateralusX]

For tools like dotnet-counters we would need to explicitly include all instruments except gc pause time, that would be "problematic" since we would need to maintain a separate list in dotnet-counters over what System.Runtime instruments to include by default.

There are also direct consumers of MeterListener (like OpenTelemetry) that would also add all instruments by default for a specific meter.

I believe the question will be, do we want existing System.Runtime subscribers to collect this automatically?

If yes, keep it in System.Runtime and address dotnet-counters presentation (maybe look into an exclude syntax). It will also affect existing usage of MeterListener, like OpenTelemetry, but it should be allowed to add instruments to existing meters, but will increase collection overhead and telemetry volumes if not explicitly filtered.

If no, put it into a different meter collecting more detailed GC instruments like histograms, but it will fragment System.Runtime meter a little, since we now have GC specific instruments scattered across meters, with no initial clear split, some of the GC instruments already in System.Runtime might have been better off put into a detailed meter in the first place.

[matyaskollert]

In build 1602112, the Windows x86 all-keywords test hit a PGO shutdown assertion. Whether this failure is independent of this PR has not yet been established.

[azure-pipelines]

Azure Pipelines:
Successfully started running 4 pipeline(s).
12 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

Unresolved moderate issues remain in histogram callback synchronization and non-multithreaded dispatcher token capture.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds an EventPipe-backed dotnet.gc.pause.duration histogram for CoreCLR and NativeAOT without introducing a public managed API.

Changes:

• Emits typed GCPause_V1 events from GC pause-accounting paths.
• Consumes events through EventListener and records histogram measurements.
• Adds lifecycle, compatibility, and regression test coverage.
• Captures per-session dispatcher cancellation tokens.

File summaries

File	Description
src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/EventPipeEventDispatcher.Threads.cs	Captures threaded dispatcher session tokens; the non-multithreaded continuation still needs equivalent capture.
src/libraries/System.Diagnostics.DiagnosticSource/tests/RuntimeMetricsTests.cs	Adds histogram, lifecycle, backlog, and dispatcher regression tests.
src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/RuntimeMetrics.GCPause.cs	Implements EventPipe-backed histogram delivery; callback recording is not synchronized with disposal and re-enablement.
src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/RuntimeMetrics.cs	Initializes the GC pause histogram.
src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/MeterListener.cs	Notifies instruments of measurement-state changes.
src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/Instrument.cs	Tracks measurement epochs and callbacks.
src/libraries/System.Diagnostics.DiagnosticSource/src/System.Diagnostics.DiagnosticSource.csproj	Includes the new GC pause source file.
src/coreclr/vm/gctoclreventsink.h	Declares CoreCLR GC event forwarding.
src/coreclr/vm/gctoclreventsink.cpp	Forwards CoreCLR GC pause events.
src/coreclr/vm/gcenv.ee.cpp	Enables GC pause-event status tracking.
src/coreclr/vm/ClrEtwAllMeta.lst	Configures event metadata.
src/coreclr/vm/ClrEtwAll.man	Defines the GC pause event schema and keyword.
src/coreclr/nativeaot/Runtime/gctoclreventsink.h	Declares NativeAOT GC event forwarding.
src/coreclr/nativeaot/Runtime/gctoclreventsink.cpp	Forwards NativeAOT GC pause events.
src/coreclr/nativeaot/Runtime/eventpipe/gen-eventing-event-inc.lst	Includes the generated GC pause event.
src/coreclr/nativeaot/Runtime/eventpipe/CMakeLists.txt	Tracks event-generation dependencies.
src/coreclr/gc/gcpriv.h	Declares pause recording support.
src/coreclr/gc/gcinternal.h	Implements pause-event emission.
src/coreclr/gc/gcinterface.h	Updates interface versions and event keywords.
src/coreclr/gc/gcinterface.ee.h	Adds the GC event-sink callback.
src/coreclr/gc/gceventstatus.h	Guards compatibility with older standalone EEs.
src/coreclr/gc/gcevents.h	Registers the GC pause event.
src/coreclr/gc/env/etmdummy.h	Adds the no-op event macro.
src/coreclr/gc/diagnostics.cpp	Emits background pause contributions.
src/coreclr/gc/collect.cpp	Emits blocking pause contributions.
src/coreclr/gc/background.cpp	Emits background GC pause contributions.

Review details

• Files reviewed: 26/26 changed files
• Comments generated: 2
• Review effort level: Lite

[lateralusX]

For tools like dotnet-counters we would need to explicitly include all instruments except gc pause time, that would be "problematic" since we would need to maintain a separate list in dotnet-counters over what System.Runtime instruments to include by default.

There are also direct consumers of MeterListener (like OpenTelemetry) that would also add all instruments by default for a specific meter.

I believe the question will be, do we want existing System.Runtime subscribers to collect this automatically?

If yes, keep it in System.Runtime and address dotnet-counters presentation (maybe look into an exclude syntax). It will also affect existing usage of MeterListener, like OpenTelemetry, but it should be allowed to add instruments to existing meters, but will increase collection overhead and telemetry volumes if not explicitly filtered.

If no, put it into a different meter collecting more detailed GC instruments like histograms, but it will fragment System.Runtime meter a little, since we now have GC specific instruments scattered across meters, with no initial clear split, some of the GC instruments already in System.Runtime might have been better off put into a detailed meter in the first place.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The documented GC pause event name/version contract conflicts with the manifest and consumers and must be reconciled.

Review effort: Lite
Findings: None

Resolved since last review (2)

• The token capture fixes only the multithreaded dispatcher. The non-multithreaded sibling still…
• The epoch check is not synchronized with the subsequent histogram.Record. A dispatcher thread can…

[jkotas]

I believe the question will be, do we want existing System.Runtime subscribers to collect this automatically?

Right. There are two parts of this question:

• Are existing System.Runtime metrics subscribers interested in this data or is it too much detail for them?
• Do we want to impose the overhead associated with collecting histograms on all System.Runtime metrics subscribes? The overhead is amplified by implementation in this PR - the implementation is simpler, at the cost of higher overhead.

The System.Runtime metrics were intentionally selected as a set that costs next to nothing to collect and that is no-brainer to enable by default and collect all the time. We have shied away from adding histograms to the default set for this reason. Relevant discussion when the System.Runtime metrics were introduced originally - #85372 (comment) .

Also, the GC pause times are not the only metric that is a valid candidate for histograms. There are other metrics like thread_pool queue length that have same characteristics as GC pause times that are candidates for histograms. If we start adding histograms to the default set, are we going to add these to the default set as well - what is the total overhead of all histograms going to be by the time we are done?

[lateralusX]

Agree. If histogram collection has a higher overhead than we want for the always-on baseline, these instruments should probably live in a separate, explicitly selected meter.

There are two costs to distinguish here: recording and aggregating the histogram, and transporting the measurements to managed code. Histograms recorded directly from managed code avoid the native-to-managed transport used here. For individual pause measurements emitted from the GC, we need a safe way to buffer and deliver that data to managed consumers.

The in-process EventListener approach reuses existing infrastructure and keeps the implementation relatively small, but comes with event transport and dispatch overhead.

The original comparison reports a producer-cost difference separately from delivery latency. Before drawing conclusions from those numbers, could we confirm whether the measured implementation included the change disabling stack capture for this event? Commit 1bf894b added that on September 18, after the measurements were posted on September 16.

The spaced-collection workload may also exercise buffer allocation more frequently than the steady-state write path: when the reader takes a writable buffer, it clears the producer's write-buffer pointer, so the next event needs a new buffer even if the previous one wasn't full. Separately, the reported ~15 ms delivery latency may largely reflect the dispatcher's explicit Thread.Sleep(10) between drain passes, including after waking on a session signal. On Windows, the commonly encountered default timer resolution of approximately 15.6 ms is consistent with that observation, although this would need confirmation in the benchmark environment.

If the collection overhead is unacceptable even for opt-in metrics, a more specialized transport may be justified. However, the earlier custom-buffer proposal still needed a managed reader thread to drain the native buffers and record the measurements into the histogram. It also needed synchronization, overflow handling, and lifecycle management, so a custom transport would not eliminate collection overhead or automatically make the histogram suitable for always-on collection.

If we invest in a custom transport (or improvements to the existing eventing infrastructure), the collection and dispatch infrastructure should be shared across native runtime instruments and optimized for that purpose. Otherwise, we risk rebuilding parts of the EventListener infrastructure separately for each instrument histogram backed by native code.

I would separate the meter-placement decision from the transport decision: both approaches could be reasonable for opt-in detailed metrics without being appropriate for the always-on System.Runtime set.

Thoughts?

[matyaskollert]

Thanks for pointing this out—the original measurements predated nostack. I reran on 93a09d8be3d, which includes it.

Average native reporting time per GC pause, on Windows x64 Release:

Requested gap between collections	Workstation	Server, 4 heaps
Back-to-back	0.18 µs	0.37 µs
10 ms	15.72 µs	14.67 µs
50 ms	42.68 µs	32.34 µs

The gaps are waits inserted by the test, outside the measured reporting call. With no consumer requesting pause events, the path only checks whether the event is enabled; its median cost was below the timer’s 0.1 µs resolution.

In the initial GCPerfSim batch, comparing histogram collection disabled versus enabled on the same build:

Measurement	Disabled → enabled	Change
Workstation throughput	7.986 → 7.583 GB/s	−5.04%
Server throughput	10.979 → 11.162 GB/s	+1.67%
Workstation CPU per GB	122.98 → 131.41 ms	+6.85%
Server CPU per GB	294.70 → 287.67 ms	−2.39%

These are preliminary local measurements, not conclusive overhead estimates. The throughput uncertainty ranges were −10.65% to +0.18% for Workstation and −3.72% to +7.25% for Server. In particular, the positive Server result should not be interpreted as a speed-up.

For spaced collections, delivery to the histogram took 14–16 ms median, with p99 around 22–26 ms. This is delayed telemetry, not additional GC pause time. The dispatcher sleep plausibly contributes, but I haven’t isolated its share.

The older custom implementation measured 3.9–6.8 µs for spaced reporting calls, but needs a matching rerun before drawing a current comparison.

So nostack does not remove the higher cost of spaced reporting. I agree these results should inform the opt-in discussion, separately from whether a custom transport is worth maintaining.

[kkokosa]

@lateralusX @jkotas - would a new opt-in System.Runtime.Histograms meter make sense for this and future histogram candidates, such as thread-pool queue length? That seems less confusing than System.Runtime.GC, since most GC metrics still remain in System.Runtime anyway. Subscribing to this meter would then be an explicit decision to accept the potential histogram collection overhead, regardless of its eventual measured size.

[jkotas]

A few thoughts:

• The primary purpose of these metrics is to expose them through OpenTelemetry. Does the OpenTelemetry project provide any guidance on how to think about histograms, given that they’re somewhat more expensive? I’d expect we’re not the first to grapple with this tradeoff. Is there any prior art for how other runtimes with garbage collectors expose GC pause durations as metrics?

• System.Runtime.Histograms: One can imagine number of metrics that are not cheap, but that are not histograms. So I would be leaving more towards names like System.Runtime.Detailed or System.Runtime.Extended.

[lateralusX]

If we believe that different runtime components might end up with instruments that are not in the default set, then maybe we should place them under their own logical meter namespace, like System.Runtime.Detailed.GC, System.Runtime.Detailed.ThreadPool, System.Runtime.Detailed.JIT etc. Then we can put cheap, always on instruments in System.Runtime meter and more resource intensive, verbose meters in the respective components detailed namespace. The instrument in this case should still be named dotnet.gc.pause.duration if we think that is the best way to describe this particular instrument.

[lateralusX]

@matyaskollert, did we get any matching numbers using the custom transport?

[steveisok]

• The primary purpose of these metrics is to expose them through OpenTelemetry. Does the OpenTelemetry project provide any guidance on how to think about histograms, given that they’re somewhat more expensive? I’d expect we’re not the first to grapple with this tradeoff. Is there any prior art for how other runtimes with garbage collectors expose GC pause durations as metrics?

Appears that go added opt-in GC metrics - open-telemetry/semantic-conventions#3629

Java and V8 appear to have similar histograms by default because they can provide it without too much noise. A separate namespace while retaining the instrument name lines up with the OTel opt-in guidance.

open-telemetry/opentelemetry-dotnet-contrib#3977 is a similar request.

[matyaskollert]

@lateralusX I've now run the same benchmarks for the custom queue too: results.

When collections were spaced apart, the custom queue took less time to record each pause and get it to the histogram. But when I forced many collections back-to-back in Workstation GC, some pause records did not arrive.

In the allocation tests, turning the custom histogram on did not show a clear overall speed difference.

These tests were run on different days, so there might be some small variations