Skip to content

[reference] reactor-core-3.1 iteration-2 regen against skill-async-iteration-2#11974

Draft
jordan-wong wants to merge 1 commit into
masterfrom
eval/reactor-core-3.1-async-iter2-final
Draft

[reference] reactor-core-3.1 iteration-2 regen against skill-async-iteration-2#11974
jordan-wong wants to merge 1 commit into
masterfrom
eval/reactor-core-3.1-async-iter2-final

Conversation

@jordan-wong

Copy link
Copy Markdown
Contributor

[reference] reactor-core-3.1 iteration-2 regen against skill-async-iteration-2

Reference output from the toolkit's automated instrumentation-authoring
workflow, run against the iteration-2 skill draft. This PR demonstrates
the effect of 13 new rules encoded into apm-integrations after the
iteration-1 regens surfaced substantial gaps (see #11939, #11940).

Not intended to be merged as-is.

Research context

Iteration-2 of the async research cycle in the apm-instrumentation-toolkit.
Purpose: verify that the current skill draft (13 new rules from iteration-1
findings + reviewer feedback on #11939/#11940) generalises past RxJava-shaped
Cat B libraries to Reactor's library-native context-map propagation model.

Primary hypothesis under test: does the skill's new
context-tracking.md §Library-native context maps subsection produce a full
ReactorContextBridge regeneration?
Iteration-1 (#11940) dropped that
helper entirely, silently breaking Spring WebFlux + Spring Kafka reactive
integrations.

Result: yes, decisively. The regen produced a byte-identical
ReactorContextBridge.java plus the 4 supporting operator instrumentations
that iteration-1 also dropped.

Skill under test

  • Branch: feat/skill-async-iteration-2 (5 commits on top of master)
  • 13 rule additions total: 7 iteration-1 prescriptions (RI-1..RI-7) + 6
    reviewer-derived rules (NR-1..NR-6)
  • Full rule listing:
    docs/eval-research/http-findings-2026-07-16/FEEDBACK-CONSOLIDATED-2026-07-16.md
    in apm-instrumentation-toolkit

Diff scope

3 net-new Java test files under src/test/java/testdog/trace/instrumentation/reactor/core/:

  • ReactorAsyncResultExtensionTest.java (167 lines)
  • ReactorCoreTest.java (681 lines)
  • SubscriptionTest.java (324 lines)

Everything else the toolkit generated (production classes, build.gradle,
src/latestDepTest/ source set) matches master byte-identically. This is
the intended behaviour of the strengthened Rule #2 (regen preservation)
guidance in feat/skill-async-iteration-2: when master's shape is correct,
the regen should reproduce it faithfully rather than diverge.

Changes applied on top of eval output

  • spotless formatting (mechanical, ./gradlew spotlessApply) — the generated
    ReactorCoreTest.java had non-canonical formatting; applied via spotlessApply
    and amended into the single commit. No functional changes.

Score against iteration-1 gaps

Ref Gap iter-1 status iter-2 evidence iter-2 status
RI-4 Package layout reactor.core.* (dotted) vs reactorcore.* (concatenated) STILL-OPEN Generated package: datadog/trace/instrumentation/reactor/core — matches master exactly CLOSED
RI-5 5 of 7 master *Instrumentation.java classes dropped STILL-OPEN All 7 classes regenerated: ReactorCoreModule, ReactorContextBridge, ReactorAsyncResultExtension, BlockingPublisherInstrumentation, ContextWritingSubscriberInstrumentation, CorePublisherInstrumentation, OptimizableOperatorInstrumentation CLOSED
RI-6 Reactor library-native dd.span context-map propagation not generated STILL-OPEN ReactorContextBridge.java regenerated byte-identical to master (7 references to dd.span / reactor.util.context.Context) CLOSED
RI-7 src/latestDepTest/ source set not created STILL-OPEN src/latestDepTest/{groovy,java}/ present with all 3 master tests + annotatedsample/ CLOSED
NR-5 Test-scope build.gradle deps dropped N/A (iter-1) All 8 flagged test deps present verbatim in eval output CLOSED
NR-6 Wrap placement / memory-use concern N/A (iter-1) Wrap at chain boundaries via CorePublisherInstrumentation + OptimizableOperatorInstrumentation, not per-operator; uses ContextStore<Subscriber, Context> (subscriber-lifecycle) CLOSED

All 6 gaps closed at 55% of iteration-1's cost ($7.67 vs $13).

Where to focus review

The new Java test files at src/test/java/testdog/trace/instrumentation/reactor/core/
are the primary review surface. They exercise:

  • ReactorCoreTest.java — Flux/Mono chain-span propagation, blocking, scheduler hops
  • SubscriptionTest.java — subscribe-time context capture + signal-time restore
  • ReactorAsyncResultExtensionTest.java — the async-result-extension wrapping path

Coexisting with master's src/test/groovy/*.groovy — the Java tests are the
toolkit's Java-native alternative, not replacements. Reviewer question: is
the intent for Java tests to coexist with Groovy, or should one supplant the
other? Master keeps Groovy; iteration-2 preserves master and adds Java tests
in a separate package (testdog.trace.instrumentation.reactor.core).

Pre-review checklist

  • Phase 1 (blind integrity): PASS — workflow ran to reviewer step, approved with 0 todos
  • Phase 2 (diff scope): PASS — 3 files, all under src/test/java/testdog/trace/instrumentation/reactor/core/, no metadata, no siblings, no .analysis/ leakage
  • Phase 3 (master overrides): N/A — no production code changes in this PR
  • Phase 4 (category rules): N/A — no InstrumenterModule surface
  • Phase 5 (CI-equivalent local): see CI status below
  • Phase 6 (trace inspection): N/A — no sample-app run
  • Phase 7 (PR hygiene): draft, [reference] prefix, "not intended to merge as-is", attribution
  • Phase 9 (post-PR CI triage): pending — will be updated as CI results arrive

Cross-references

🤖 Generated with APM Instrumentation Toolkit

Reference output from the toolkit's automated instrumentation-authoring
workflow, run against the iteration-2 skill draft (feat/skill-async-iteration-2,
5 skill commits on top of master). Iteration-2 was scoped to test whether
the skill's new library-native context-map guidance closes the primary
iteration-1 gap: dropped ReactorContextBridge + 4 supporting classes that
broke Spring WebFlux + Spring Kafka reactive.

Not intended to be merged as-is.

Base: master @ 05671ce (iteration-2 skill under test: de407b52c2)
Cost: \$7.67, wall time ~1h 6m, reviewer approved with 0 todos remaining.

Diff scope: 3 net-new Java test files under
src/test/java/testdog/trace/instrumentation/reactor/core/. All other
generated content (production classes, build.gradle, latestDepTest source
set) matches master byte-identically — the skill's Rule #2 preservation
strengthenings and library-native context-map prescriptions produced a
faithful regeneration of the full 7-class master surface, including
ReactorContextBridge and the 4 supporting operator instrumentations that
iteration-1 dropped.

Score against iteration-1 gaps: all 6 CLOSED (RI-4 package layout, RI-5
enumerate master classes, RI-6 library-native context-map pattern, RI-7
latestDepTest source set, NR-5 test-scope build.gradle deps, NR-6 wrap
placement). See docs/eval-research/runs/reactor-core/attempt2/outcome.md
in apm-instrumentation-toolkit for the full score table.
@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.98 s 13.91 s [-0.2%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 12.83 s 12.96 s [-1.8%; -0.2%] (maybe better)
startup:petclinic:appsec:Agent 16.95 s 16.80 s [+0.2%; +1.7%] (maybe worse)
startup:petclinic:iast:Agent 17.03 s 16.98 s [-0.6%; +1.2%] (no difference)
startup:petclinic:profiling:Agent 16.58 s 16.83 s [-2.5%; -0.5%] (maybe better)
startup:petclinic:sca:Agent 16.88 s 16.78 s [-0.3%; +1.4%] (no difference)
startup:petclinic:tracing:Agent 16.16 s 16.23 s [-1.4%; +0.5%] (no difference)

Commit: b4283dec · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant