feat(nativemem): categorized native-memory accounting — first cut#669
Draft
rkennke wants to merge 2 commits into
Draft
feat(nativemem): categorized native-memory accounting — first cut#669rkennke wants to merge 2 commits into
rkennke wants to merge 2 commits into
Conversation
Add a NativeMem facility that tracks the profiler's own native memory
usage per category, with a moving-window average and a running peak.
- NativeMemCategory enum whose per-category live gauges partition the
total (each backing allocation belongs to exactly one category, so
there is no double counting).
- Live accounting is always-on and independent of the COUNTERS build
flag: record() is a single relaxed atomic add, async-signal-safe and
usable from signal handlers.
- sample() folds the live gauges into a moving-window average and a
high-water max; it is ticked once per JFR chunk finish.
- Totals mirror into the existing NATIVE_MEM_{LIVE,AVG,MAX}_BYTES
counters (JFR + JNI counter path); per-category values are emitted as
native_mem_{live,avg,max}_bytes.<category> counter events, reusing the
existing counter event format (no new event type).
Instrumented sites (tagged CALLTRACE, their sole use today): the
LinearAllocator chunk alloc/free (the call-trace arena) and the
per-shard calltrace buffers. Accounting lives with the semantic owner
rather than OS::safeAlloc, which stays category-agnostic.
First-cut limits, documented in code: only CALLTRACE sites are
instrumented so far (other categories read 0 until tagged); the max is
sampled rather than spike-accurate; transient negative live is clamped
to 0. The existing reserved/used/waste counters (CALLTRACE_STORAGE_BYTES,
DICTIONARY_ARENA_WASTE_BYTES) remain an independent nested dimension and
are intentionally not summed into the per-category total.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
CI Test ResultsRun: #29589359986 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-07-17 15:06:13 UTC |
Contributor
Benchmark Results (commit e095203)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125289326 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10287 ms (21 iters) | ✅ 10260 ms (21 iters) | ≈ -0.3% (±10.9%) | — / — |
| akka-uct | 25 | ✅ 8867 ms (24 iters) | ✅ 8805 ms (24 iters) | ≈ -0.7% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 5933 ms (33 iters) | ✅ 5935 ms (33 iters) | ≈ +0% (±24.9%) | |
| finagle-chirper | 25 | ✅ 5475 ms (36 iters) | ✅ 5453 ms (36 iters) | ≈ -0.4% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2648 ms (71 iters) | ✅ 2700 ms (69 iters) | ≈ +2% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2825 ms (66 iters) | ✅ 2836 ms (66 iters) | ≈ +0.4% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2121 ms (88 iters) | ✅ 2045 ms (90 iters) | 🟢 -3.6% | — / — |
| future-genetic | 25 | ✅ 1976 ms (94 iters) | ✅ 2086 ms (89 iters) | 🔴 +5.6% | — / — |
| naive-bayes | 21 | ✅ 1230 ms (139 iters) | ✅ 1259 ms (137 iters) | ≈ +2.4% (±32.9%) | — / — |
| naive-bayes | 25 | ✅ 1015 ms (168 iters) | ✅ 1011 ms (169 iters) | ≈ -0.4% (±31.6%) | — / — |
| reactors | 21 | ✅ 15811 ms (16 iters) | ✅ 16445 ms (15 iters) | ≈ +4% (±7.6%) | — / — |
| reactors | 25 | ✅ 18243 ms (15 iters) | ✅ 18764 ms (15 iters) | ≈ +2.9% (±4.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2094 / 1933 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2216 / 2195 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 8790 / 8777 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 8165 / 8254 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1283 / 1240 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1289 / 1281 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2935 / 2926 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 2884 / 2927 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 2 | 3486 / 3580 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / ✅ | 3448 / 3533 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1512 / 1844 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1866 / 1889 | ✅ / ✅ | ✅ / ✅ |
This comment has been minimized.
This comment has been minimized.
Contributor
Reliability & Chaos Results❌ 2 failure(s) detected Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/125289287 ❌ chaos: profiler tcmalloc amd64 21 0 3 temXchaos❌ chaos: profiler tracer tcmalloc aarch64 25 0 3 temXchaos |
Contributor
Benchmark Results (commit 62b6f2c)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125313150 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10303 ms (21 iters) | ✅ 10356 ms (21 iters) | ≈ +0.5% (±10.7%) | — / — |
| akka-uct | 25 | ✅ 9028 ms (24 iters) | ✅ 8895 ms (24 iters) | ≈ -1.5% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 5921 ms (33 iters) | ✅ 5936 ms (33 iters) | ≈ +0.3% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5537 ms (36 iters) | ✅ 5535 ms (36 iters) | ≈ -0% (±24.4%) | |
| fj-kmeans | 21 | ✅ 2705 ms (70 iters) | ✅ 2667 ms (69 iters) | ≈ -1.4% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2829 ms (66 iters) | ✅ 2824 ms (66 iters) | ≈ -0.2% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2122 ms (88 iters) | ✅ 2066 ms (90 iters) | 🟢 -2.6% | — / — |
| future-genetic | 25 | ✅ 2071 ms (90 iters) | ✅ 1960 ms (94 iters) | 🟢 -5.4% | — / — |
| naive-bayes | 25 | ✅ 1022 ms (167 iters) | ✅ 1012 ms (169 iters) | ≈ -1% (±31.6%) | — / — |
| reactors | 21 | ✅ 16191 ms (15 iters) | ✅ 15952 ms (15 iters) | ≈ -1.5% (±6.7%) | — / — |
| reactors | 25 | ✅ 18581 ms (15 iters) | ✅ 18426 ms (15 iters) | ≈ -0.8% (±3.8%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / ✅ | 1975 / 2029 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2406 / 2283 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 7 / 7 | 8749 / 8443 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 8539 / 8755 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 1305 / 1244 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 1284 / 1292 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 3065 / 2961 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 2938 / 2818 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 5 / 4 | 3483 / 3485 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1495 / 1554 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1874 / 1984 | ✅ / ✅ | ✅ / ✅ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a
NativeMemfacility to measure the profiler's own native memory usage — currently we only observe whole-process RSS. This is the first cut of a phased effort: get a per-category live gauge plus a moving-window average and a running max, wired through the existing counter/JFR path.Why
We measure the agent's native footprint only as an RSS delta (~150–200 MB) with no breakdown. This gives us an in-process, attributable number we can categorize and trend, and a foundation for later identifying which code is responsible.
How
NativeMem(nativeMem.h/nativeMem.cpp): aNativeMemCategoryenum whose per-category live gauges partition the total — each backing allocation belongs to exactly one category, so there's no double-counting.COUNTERSbuild flag:record()is a single relaxed atomic add, which is async-signal-safe (usable from signal handlers).sample()folds the live gauges into a moving-window average and a high-water max, ticked once per JFR chunk finish (Recording::finishChunk).NATIVE_MEM_{LIVE,AVG,MAX}_BYTEScounters (JFRT_DATADOG_COUNTER+ JNI debug counters); per-category values are emitted asnative_mem_{live,avg,max}_bytes.<category>events reusing the same event format — no new event type or schema change.CALLTRACE, their sole use today):LinearAllocatorchunk alloc/free (the call-trace arena) and the per-shard calltrace buffers. Accounting lives with the semantic owner rather than inOS::safeAlloc, which stays category-agnostic.Avoiding double-counting
The call-trace bytes are visible at three layers today —
safeAllocmmap,LINEAR_ALLOCATOR_BYTES(reserved chunks), andCALLTRACE_STORAGE_BYTES(used within chunks). Summing counters would double/triple-count. The new aggregate counts backing memory once per category; the existing reserved/used/waste counters (CALLTRACE_STORAGE_BYTES,DICTIONARY_ARENA_WASTE_BYTES) remain an independent nested dimension and are intentionally not summed into the per-category total.First-cut limitations (documented in code)
CALLTRACEsites are instrumented; other categories exist but read 0 until theirmalloc/calloc/newsites are tagged (or a future malloc-interception pass captures them). Expect the total to be a correct subset of the RSS delta.Roadmap
CALLTRACEcoverage + total/per-category avg/max.Testing
nativeMem_ut(6 tests): live tracking + total partition, single-sample avg==live, moving average over ticks, max high-water behavior, negative-live clamping, category-name presence.:ddprof-lib:gtestDebugsuite green — 335 tests, 0 failures.🤖 Generated with Claude Code