Store known span tags densely in TagMap by tag-id

[dougqh]

Draft for preservation — not for merge yet. Per plan: lands after next week's release, with both the read-through and dense pieces behind experimental flags (off by default, flipped on internally to bake).

This branch is stacked: read-through (level-split) commits + the dense known-tag store on top.

Read-through (level-split phase 1) — stacked, tracked separately

The mechanism is draft #11789. This branch also carries the consumer wiring (mergedTracerTags as a read-through parent) + benchmark; those will be split into the read-through PR(s), which land first. Still TODO: add an experimental flag gating the read-through parent wiring (currently gated only on !needsIntercept).

Dense known-tag store (this PR's focus)

Known tags (those KnownTags.keyOf resolves to a stored id) store in dense parallel arrays (knownIds/knownValues, insertion-ordered, cap-8 ×2) with no per-tag Entry — the #1 tracer allocator. Coexists with the hash buckets (arbitrary tags); disjoint by global known-ness, so read-through shadow checks stay within-region.

• keyOf substrate — KnownTags (tagId encoding) + KnownTagIds (registry + open-addressed keyOf via StringIndex)
• dense store woven into OptimizedTagMap (set/get/remove/forEach/size/copy/putAll/clear/iterate)
• lazy buckets — shared EMPTY_BUCKETS sentinel + copy-on-write; all-known maps allocate zero buckets
• flyweight EntryReader iterator — alloc-free dense reads on the serialize path; entrySet() keeps real retain-safe Entry
• dormant behind dd.trace.dense.tags.enabled (experimental system property; → promote to a Config flag before ship)

Validation

• Deterministic -prof gc (real resolver, realistic mixed span): per-build alloc −8% (7 tags) → −41% (12 tags); all-known / trace-tier shape −57%/−43%.
• PetClinic macro (Xmx96m, counterbalanced A-B-B-A): GC pauses −2.8% (vs +6.6% before the flyweight fix), throughput break-even.
• Coverage: OptimizedTagMapDenseForkedTest (real registry + read-through union), TagMapDenseFuzzForkedTest (3 key regimes, HashMap oracle + checkIntegrity), DenseStoreAllocBenchmark.

The CPU upside (skip keyOf by setting tags via setTag(KnownTagIds.X)) arrives via incremental instrumentation migration over time — no big-bang.

🤖 Generated with Claude Code

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.96 s	13.83 s	[+0.1%; +1.7%] (maybe worse)
startup:insecure-bank:tracing:Agent	12.93 s	13.04 s	[-1.6%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent	16.85 s	16.73 s	[-0.3%; +1.8%] (no difference)
startup:petclinic:iast:Agent	16.95 s	16.47 s	[-1.5%; +7.2%] (no difference)
startup:petclinic:profiling:Agent	16.75 s	16.36 s	[-2.3%; +7.0%] (no difference)
startup:petclinic:sca:Agent	16.90 s	16.70 s	[+0.2%; +2.1%] (maybe worse)
startup:petclinic:tracing:Agent	16.14 s	16.18 s	[-1.1%; +0.7%] (no difference)

Commit: 6c708481 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[dougqh]

Directional span-creation benchmark: dense off vs on (this branch)

Toggle A/B on this branch (-Ddd.trace.dense.tags.enabled), so it isolates the dense store from the rest of the stack — byte-identical code, just the flag. Grafted SpanCreationBenchmark (from #11915), not committed.

Config: -f 2 -wi 3 -i 3 -w 5 -r 5 -prof gc, @Threads(8), JDK 17. Realistic trace bundle via -jvmArgsAppend "-Ddd.env=bench -Ddd.version=1.0.0 -Ddd.tags=team:apm,tier:web,region:us1,shard:7,build:1234". Alloc = gc.alloc.rate.norm. Without bloom (#11900); alloc is bloom-independent, but throughput here reflects the slower linear-scan insertion — directional only.

method	off B/op	on B/op	Δ% (cap 12)	Δ% (cap 8)
bareStartSpan	52,429	52,779	+0.67%	+0.78%
bareBuildSpan	52,403	52,879	+0.91%	+0.65%
webServerSpan	62,800	63,156	+0.57%	+0.23%
webServerSpanViaBuilder	69,306	70,334	+1.48%	+1.25%
jdbcClientSpan	64,370	64,677	+0.48%	+0.75%

Honest read: dense in isolation is alloc-negative here (~+0.5–1.5%)

This is a local valley, not the destination — and the cause is instructive:

• It's over-provisioning, not array growth. The regression is cap-insensitive — bumping KNOWN_INIT_CAP 8→12 shuffles within run noise and the worst case stayed worst. If growth were the driver, cap-12 (≥ every span's known count) would have clearly rescued the exit spans. It didn't. The cost is the fixed two-array allocation (long[]+Object[]) per span-with-known-tags; at real span tag counts that exceeds the Entry bytes eliminated, regardless of cap. So a stopgap size can't fix it.

• The win is jointly gated on two upstack pieces:

  1. Per-type exact sizing (tag registry) — a webServer span sized to its ~6 own known tags (long[6]+Object[6] ≈ 104 B) genuinely beats ~6 Entries (≈ 190 B). Generous fixed caps over-provision that away.
  2. Level-split actually engaging — the trace bundle (env/version/…) is landing in every span's dense array (even bare spans regress), because an intercepted tag in mergedTracerTags gates read-through off and forces a copy. Stripping it out via read-through shrinks the per-span known count.

Reconciles with the macro TagMap$Entry→0 finding: dense removes the Entries but adds array bytes; the net only turns positive once the arrays are packed tight (registry) and the trace bundle is read through (level-split). Sizing is a generous fixed stopgap (KNOWN_INIT_CAP=12) until the registry lands.

From Claude: directional A/B run during the dense reconcile onto the level-split stack; posting the honest before/after and the coupled-gating that gets us out of the valley.

🤖 Generated with Claude Code