Skip to content

fix: give net/http, Prometheus, OTel and slog copies of the request strings they keep, and give Adapt's request its headers (#732, #720) - #736

Merged
FumingPower3925 merged 7 commits into
mainfrom
fix/celeris-732-720-retained-views
Sep 27, 2026
Merged

FumingPower3925 merged 7 commits into
mainfrom
fix/celeris-732-720-retained-views

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #720. Refs #732: this fixes four of its sites; #742 tracks the rest (a fifth site in the recovery middleware, and otel SLICE/MAP custom attribute values), so #732 stays open.

What was wrong

On epoll and io_uring, request strings (method, path, query, Host, header names and values, and whatever middleware derive from them) are views of the connection's receive buffer. The engine reuses that buffer for the connection's next request, and once the connection closes it pools the buffer and a newly accepted connection reads into it. Reading a view during the request is fine. Keeping one after the request is not, because the kept string later reads whatever the engine receives next.

Four sites handed these views to APIs that keep them (#732):

  • celeris.Adapt and adapters.WrapMiddleware built the *http.Request from the views. net/http lets a handler or middleware keep the request's strings after ServeHTTP returns, for example in a map key, a queued log line or a goroutine.
  • metrics passed c.Method() (for a method the H1 parser does not intern), c.Path() (when there is no route pattern) and every LabelFuncs value to WithLabelValues. client_golang keeps the label values of every new series for the life of the registry. The series' labels then changed to later request bytes: Gather reported duplicate series, and one requests_total label read ion: SECRET, bytes of another connection's Authorization: SECRET... header.
  • otel built span attributes (url.path, server.address, user_agent.original, client.address, request.id, url.scheme, http.request.method_original, the CustomAttributes values, and the span name when SpanNameFormatter is set) and metric attributes (server.address, url.scheme, CustomMetricAttributes) from the views. normalizeMethod also returned its argument, so http.request.method for TRACE and CONNECT was a view too, since the H1 parser does not intern those methods. A span processor keeps an ended span until it exports it, and the metric SDK keeps every attribute set as an aggregation key.
  • logger logged the views (method, path, Host, User-Agent, Referer, query, client IP, request ID, and LogContextKeys, LogResponseHeaders and Fields values derived from headers). slog lets a handler keep a record after Handle returns by calling Record.Clone, which shares the strings. Asynchronous and batching handlers do this.

Separately (#720), buildHTTPRequest read c.stream.Headers without calling MaterializeHeaders. The native H1 parser fills that slice only when something reads a header, so on a route with no earlier header read, the adapted handler got a request with no headers. This also meant adapters.ReverseProxy, which is built on Adapt, forwarded no request headers on epoll and io_uring. std and adapters.WrapMiddleware were not affected.

The fix

site change copies
Adapt (bridge.go) c.stream.MaterializeHeaders() before the header loop (#720). Method, URL, header names and values, and Host are copied into one buffer and cut back out in order. 1 allocation per request. The "?" concatenation is gone, because path and query are one copied string.
adapters.buildRequest Same one-buffer copy. 1 allocation per request.
metrics The middleware keeps a set of the label combinations it has recorded, keyed by the values (length-prefixed, built on the stack). A new combination is copied once into the map key, and the values handed to Prometheus are cut from that copy. The set also keeps the resolved series, so a combination seen before copies nothing and makes one map lookup instead of four WithLabelValues calls. The size histograms are still resolved on first use, so a combination that never carried a body still has no request_size_bytes series. WithLabelValues runs outside the set's lock because it panics on invalid UTF-8. Only for a new label combination.
otel ownStrings copies the request strings the span and metric attributes use into one allocation. normalizeMethod returns its own constants. String and string-slice values from CustomAttributes and CustomMetricAttributes are copied as they are appended (keys are not). 1 allocation per request, plus 1 per custom string attribute.
logger At the handler boundary, a handler other than the package's own (FastHandler and its WithGroup handler, which format before they return) gets copies of every string value, including values inside groups. FastHandler is unchanged. 1 allocation per logged request, only for handlers other than FastHandler.

The request body is not copied. net/http forbids reading r.Body after ServeHTTP returns.

One difference from net/http remains: with the headers now present, the adapted request on epoll and io_uring also carries Host in r.Header, as adapters.WrapMiddleware's already did. net/http's own server moves it to r.Host only. r.Host is set in both cases.

Tests

Each test has five arms: std, and epoll and io_uring with sync and async (AsyncHandlers) handlers. Requests with the same layout and different values go over one keep-alive connection, and what the API kept from an earlier request must still read that request.

test tests on main, native arms + the #720 commit only head
TestAdaptRequestCarriesHeaders (#720), root FAIL 4/4 (0 headers; std PASS) PASS 5/5 PASS 5/5
TestAdaptKeptRequestStringsSurviveNextRequest, root FAIL 4/4 (12 wrong fields, e.g. request 1 r.URL.Path = /a/cccc, r.Method = MKCOL) FAIL 4/4 PASS 5/5
TestWrapMiddlewareKeptStringsSurviveNextRequest, middleware/adapters FAIL 4/4 (10 wrong) FAIL 4/4 PASS 5/5
TestLabelValuesSurviveNextRequest, middleware/metrics FAIL 4/4 (1 of 3 series left in each metric, 6 Gather errors) FAIL 4/4 PASS 5/5
TestLabelValuesSurviveConnectionReuse, middleware/metrics FAIL on epoll and io_uring (a label read ion: SECRET from another connection; 42 and 36 Gather errors). PASS on the two async arms, see below same PASS 5/5
TestAttributesSurviveNextRequest, middleware/otel FAIL 4/4 (19 wrong, e.g. span 1 url.path = /o/cccc, http.request.method = MKCOL) FAIL 4/4 PASS 5/5
TestKeptRecordSurvivesNextRequest, middleware/logger FAIL 4/4 (22 wrong) FAIL 4/4 PASS 5/5

"Tests on main" is the test commit alone, run on a842109 and again after rebasing onto 2776d4c (#696 landed meanwhile), with the same results. std passes every test at every commit. On main, 26 of the 28 native arms fail. The two that pass are the cross-connection test's async arms: with async handlers the request is parsed from the dispatch double buffer (asyncInBuf), and in this layout no series read the other connection's bytes in 20 rounds. They stay as coverage. The same-connection metrics test covers async handlers.

Runs used Docker linux/arm64 in the CI shape (golang:1.27, --cpus 4, seccomp=unconfined, memlock 8 MiB, so io_uring gets one worker), -race, one container per package, and CELERIS_REQUIRE_IOURING_WORKERS=1, so no io_uring arm could be dropped. Counts are tallied from --- PASS/FAIL/SKIP: lines only.

Mutants. There were 31 mutants, each re-introducing one view or dropping the materialize call: 6 in Adapt, 5 in WrapMiddleware, 3 in metrics (each routes one metric around the set with the request's strings), 13 in otel (one per copied field, normalizeMethod, and the custom-attribute copies), and 4 in logger. All 31 were killed. A mutant counted as killed only if its test failed on all four native arms while std passed. Each mutant was restored with cp, and the restore was proved by sha256. One mutant did not compile on its first version (m unused); it was rewritten, killed, and the invalid log kept. The mutants and the benchmarks below ran before the rebase onto 2776d4c. The rebase changed no commit's patch (git range-diff), and #696 touches only the io_uring driver-conn path, none of these files or the HTTP request path. The mutants' head differs from this one only in six defer conn.Close() lines of the tests, rewritten for errcheck.

Whole-package suites (-race -v) at head (root rerun after the rebase): root 368 passed, 0 failed (the 1 top-level and 4 nested SKIPs are the pre-existing adaptive retime tests); middleware/adapters 23, logger 116, metrics 47, otel 63 passed, 0 failed, 0 skipped. golangci-lint v2.13 (GOOS=linux) reports 0 issues on the changed packages, and actionlint passes.

CI. The package steps already run these packages, but without -v, and each test drops its io_uring arms silently when the probe finds no ring. A new Unit step runs the seven tests by name with CELERIS_REQUIRE_IOURING_WORKERS=1 and an exact tally: 7 top-level tests, 35 arms (14 io_uring), no SKIP line, and every go test must exit 0.

Cost (per request)

Benchmarks run each site through a test Context (celeristest, no engine), with a typical API request's headers. Base is a842109 and head is this branch, built as test binaries and run interleaved A B A B for 8 rounds in one container (golang:1.27 linux/arm64, --cpus 4) under the laptop's exclusive timing lock. B/op and allocs/op are exact. ns/op comes from a shared laptop. Δ ns is the median of the 8 per-round (head − base) differences.

benchmark base ns head ns Δ ns (range) B/op allocs/op
Adapt 828 901 +96 (−86..+203) 1377 → 1633 15 → 16
adapters.WrapMiddleware (pass-through) 902 1039 +134 (−68..+287) 1673 → 1930 19 → 20
metrics, combination seen before 376 256 −117 (−154..+19) 64 → 64 1 → 1
metrics, seen before, with a LabelFuncs label 429 262 −166 (−191..−127) 64 → 64 1 → 1
metrics, every request a new combination 2180 2331 +82 (−240..+1501) 3939 → 4217 63 → 67
otel, SDK tracer and meter 3351 3233 −33 (−1152..+102) 4947 → 5027 20 → 21
otel, no-op providers 1282 1383 +52 (−61..+209) 2417 → 2497 15 → 16
logger, FastHandler 290 292 −6 (−33..+16) 80 → 80 4 → 4
logger, slog.JSONHandler 1111 1299 +150 (+35..+215) 400 → 560 5 → 6
trivial handler, no middleware (same code both arms: the noise floor) 233 238 +12 (−24..+19) 256 1

A trivial handler's per-request time depends on what is counted:

  • Handler only, in process (233 ns; no parsing and no syscalls): every site that copies costs more than 1%. Adapt is +41%, WrapMiddleware +58%, otel with no-op providers +22%, and logger with a handler other than FastHandler +64%. A new metrics label combination is +35%, paid once per combination.
  • One request as a client sees it (25.7 µs median round trip, one keep-alive loopback connection on epoll, same container): every site is under 1%. The largest is the logger with slog.JSONHandler at +0.6%.

Each copying site adds one allocation per request (plus one per custom otel string attribute). metrics now costs less than before on every request whose label combination was seen before. FastHandler and the OTel SDK path show no change beyond the noise. A cheaper logger option would skip the copies for slog.JSONHandler/slog.TextHandler, which format before they return. It is left out because their ReplaceAttr callback can keep a value; FastHandler users pay nothing.

Overlap

…rings kept by net/http, Prometheus, OTel and slog survive the next request (#720, #732)

Each test runs std, and epoll and io_uring with sync and async handlers.
Requests with the same layout and different values go over one keep-alive
connection (for the metrics cross-connection test, over a closed
connection's pooled buffer), and what the API kept from an earlier request
must still read that request:

- TestAdaptRequestCarriesHeaders (#720): the adapted handler's request has
  the request headers, with nothing reading a header before Adapt.
- TestAdaptKeptRequestStringsSurviveNextRequest, and
  TestWrapMiddlewareKeptStringsSurviveNextRequest in middleware/adapters:
  the method, path, query, Host, a header value and a header name net/http
  does not canonicalize, kept by the net/http handler or middleware.
- TestLabelValuesSurviveNextRequest and TestLabelValuesSurviveConnectionReuse
  in middleware/metrics: the requests_total series labels (method, a
  LabelFuncs value), with no Gather error, and no label holding another
  connection's Authorization bytes.
- TestAttributesSurviveNextRequest in middleware/otel: the span name and
  string attributes and the duration series attributes.
- TestKeptRecordSurvivesNextRequest in middleware/logger: every string
  attribute of a record a handler keeps with Record.Clone.

A CI step runs the seven tests by name with CELERIS_REQUIRE_IOURING_WORKERS=1
and an exact tally (7 tests, 35 arms, 14 of them io_uring, no SKIP), since
the package steps run without -v and a test drops its io_uring arms silently
when the probe finds no ring.
…ring (#720)

buildHTTPRequest copied the headers from c.stream.Headers without calling
MaterializeHeaders. The H1 parser of the native engines fills that slice
only when something reads a header (populateCachedStream keeps the raw
headers in LazyRawHeaders), so on a route with no header read before Adapt
the adapted handler got a request with no headers: no Authorization,
Cookie or Content-Type. This also affected adapters.ReverseProxy, which is
built on Adapt and so forwarded no request headers. std fills the slice
eagerly, and middleware/adapters reads it through RequestHeaders, which
materializes it; neither was affected.
…s.WrapMiddleware (#732)

net/http lets a handler or middleware keep the request's strings after
ServeHTTP returns: a log line queued for later, a rate limiter's map key, a
value handed to a goroutine. buildHTTPRequest (Adapt, AdaptFunc and
adapters.ReverseProxy) and middleware/adapters.buildRequest
(WrapMiddleware) built the request from the method, path, query, Host and
header strings as they were: on epoll and io_uring, views of the
connection's receive buffer, which the engine reuses for the connection's
next request and, once the connection closes, for another connection. A
kept string then read other request bytes.

Both now copy every string they hand to net/http into one allocation per
request, cut back out in order, so the URL (path and query) is one string
and the "?" concatenation is gone. Header names are copied too: net/http
canonicalizes a name by allocating a new string, except when it is already
canonical (for example, a name with no letters), which it returns as it
is. The body is not copied: net/http forbids reading it after ServeHTTP
returns.
…er new label combination (#732)

client_golang keeps the label values of every new series for the life of
the registry and does not copy them. The middleware passed the request's
strings: c.Method() for a method the H1 parser does not intern, c.Path()
when there is no route pattern, and every LabelFuncs value (typically a
header). On epoll and io_uring those are views of the connection's receive
buffer, which the engine reuses for the connection's next request and, once
the connection closes, for another connection, so a series' labels changed
to other request bytes: Gather reported duplicate series, and a label read
another client's Authorization header. (The label-value pool's comment said
Prometheus does not retain the values; it keeps them when a series is new.)

The middleware now keeps its own set of the label combinations it has
recorded, keyed by the values (length-prefixed, built on the stack). A
combination seen for the first time is copied once, into the map key, and
the values handed to Prometheus are cut from that copy. The set also keeps
the series it resolved: a request whose combination was seen before copies
nothing and makes one map lookup instead of four WithLabelValues calls (each
of which validates, hashes and looks up the values under the vector's
lock). The request and response size histograms are resolved on first use,
so a combination that never carried a body still has no size series.
WithLabelValues runs outside the set's lock, since it panics on a label
value that is not valid UTF-8.
…ings (#732)

A span processor keeps an ended span until it exports it, and the metric
SDK keeps every attribute set it has seen as an aggregation key for the
life of the provider; neither copies strings. The middleware built them
from the request's strings: url.path, server.address, user_agent.original,
client.address, request.id, url.scheme, http.request.method_original, the
span name when SpanNameFormatter is set or there is no route, and the
CustomAttributes and CustomMetricAttributes values. On epoll and io_uring
those are views of the connection's receive buffer, which the engine reuses
for the connection's next request and, once the connection closes, for
another connection, so a kept span or attribute set read other request
bytes.

The request strings are now copied into one allocation per request
(ownStrings), and the span and metric attributes share the copies.
normalizeMethod returns the package's own constant instead of its argument,
so http.request.method for TRACE and CONNECT (standard, but not interned by
the H1 parser) is no longer a view. String and string-slice values of the
custom attributes are copied as they are appended; keys are not.

Only otel.go and config.go change; carrier.go (the propagator carrier, in
PR #723) is untouched.
…string values (#732)

slog lets a Handler keep a Record after Handle returns by calling
Record.Clone, which shares the strings; asynchronous and batching handlers
do. The middleware logged the request's strings: method, path, Host,
User-Agent, Referer, query, client IP, request ID, and whatever
LogContextKeys, LogResponseHeaders and Fields values code derived from
request headers. On epoll and io_uring those are views of the connection's
receive buffer, which the engine reuses for the connection's next request
and, once the connection closes, for another connection, so a kept record
later formatted other request bytes.

At the handler boundary, a handler other than the package's own (FastHandler
and its WithGroup handler, which format the record before they return) now
receives copies of every string value, descending into groups; the
top-level copies share one allocation per logged request. FastHandler's
path is unchanged and copies nothing.
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Sep 27, 2026
@FumingPower3925 FumingPower3925 added bug Something isn't working area/observability Metrics, logging, debug endpoints area/api Public-facing API surface security Security hardening middleware Middleware implementation labels Sep 27, 2026
@coderabbitai

coderabbitai Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: goceleris/celeris/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: cdeea9a2-7140-425d-8019-18da4b6e5caf

📥 Commits

Reviewing files that changed from the base of the PR and between 7b6cdca and 30c5c41.

📒 Files selected for processing (1)
  • .github/workflows/ci.yml

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.


📝 Walkthrough

Walkthrough

The pull request materializes headers and copies request-derived strings before adapted handlers and middleware can retain them. Linux tests check header availability and string retention across requests, connections, and supported engines. CI runs the retention tests with race detection.

Changes

Request String Retention

Layer / File(s) Summary
Request conversion boundaries
bridge.go, adapt_request_linux_test.go, middleware/adapters/adapters.go, middleware/adapters/retained_strings_linux_test.go
buildHTTPRequest materializes deferred headers and copies request strings. buildRequest also uses copied request strings. Linux tests check header availability and retained values across requests.
Logger record ownership
middleware/logger/config.go, middleware/logger/logger.go, middleware/logger/retained_attrs_linux_test.go
Non-fast handlers receive records with copied string values, including values in groups. Linux tests check retained log attributes.
Metrics series ownership
middleware/metrics/config.go, middleware/metrics/metrics.go, middleware/metrics/retained_labels_linux_test.go
Metrics caches label combinations and instruments, and retains owned strings for new combinations. Linux tests check series across reused requests and connections.
OpenTelemetry attribute ownership
middleware/otel/config.go, middleware/otel/otel.go, middleware/otel/retained_attrs_linux_test.go
OpenTelemetry middleware copies request-derived and custom string attributes for spans and metrics. Linux tests check retained values.
CI retention checks
.github/workflows/ci.yml
CI runs seven retention tests with race detection. It fails if test commands or test arms fail, if tests are skipped, or if the expected arm counts are not met.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 30c5c

No actionable merge-blocking risk is identified. The change is mergeable after normal checks.

Architecture Summary

Architecture risk: 🔵 Low · up to 30c5c

The change affects 3 systems.

Changed systems: middleware, adapt_request_linux_test.go, bridge.go

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — middleware (service) was modified; 11 changed files map to changed impact.
  • observed — adapt_request_linux_test.go (service) was modified; 1 changed file maps to changed impact.
  • observed — bridge.go (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in adapt_request_linux_test.go: Added Linux-only TestAdaptRequestCarriesHeaders, which checks that adapted handlers receive four specified request headers across the engine and handler-mode arms returned by keptArms.
  • observed — Modified behavior in adapt_request_linux_test.go: Added TestAdaptKeptRequestStringsSurviveNextRequest, which sends three requests with distinct methods and values over one connection, retains selected request strings in the handler, and compares them with their expected per-request values.
  • observed — Modified behavior in adapt_request_linux_test.go: Added keptCompare to read one retained-value map per expected request, fail if a report takes over 10 seconds, and report mismatched fields.
  • observed — Modified behavior in adapt_request_linux_test.go: Added the engine-arm configuration and selection: standard and epoll arms always run; io_uring arms run when the probe reports tier High or above with provided buffers. When CELERIS_REQUIRE_IOURING_WORKERS=1, unavailable io_uring support fails the test; otherwise those arms are logged as skipped.
🚥 Pre-merge checks | ✅ 2 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Title check ⚠️ Warning The title uses the required fix: prefix and accurately describes the changes, but its issue references use (#732, #720) instead of the required repository-qualified format such as (celeris#720). End the title with repository-qualified references, for example (celeris#732, celeris#720). If only one issue is the fix target, use its reference last.
Out of Scope Changes check ⚠️ Warning The changes in middleware/adapters, middleware/logger, middleware/metrics, and middleware/otel implement retained request-string ownership. Their tests and documentation cover that behavior. T… Remove the retained-string changes from this pull request, or provide a directly linked active issue that requires them.
✅ Passed checks (2 passed)
Check name Status Explanation
Description check ✅ Passed The description directly explains the retained request-string defects, missing Adapt headers, implementation changes, tests, benchmarks, and issue scope.
Linked Issues check ✅ Passed #720 requires celeris.Adapt to provide request headers on epoll and io_uring without prior header access. bridge.go now calls MaterializeHeaders() before iterating headers. `adapt_request_linux_…
Full details: Out of Scope Changes check

Explanation

The changes in middleware/adapters, middleware/logger, middleware/metrics, and middleware/otel implement retained request-string ownership. Their tests and documentation cover that behavior. The linked issue #720 concerns only missing headers in celeris.Adapt; these changes have no demonstrated connection to that objective.

  • Fix all pre-merge checks with AI

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Sep 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.13514% with 9 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
middleware/otel/otel.go 90.00% 5 Missing ⚠️
bridge.go 91.66% 3 Missing ⚠️
middleware/metrics/metrics.go 97.77% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@codspeed

codspeed Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 2.12%

⚡ 3 improved benchmarks
❌ 6 regressed benchmarks
✅ 56 untouched benchmarks
⏩ 5 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ BenchmarkContextQueryFirstParse 736 ns 904 ns -18.58%
❌ BenchmarkChainBaseline 1.9 µs 2.3 µs -16.75%
❌ BenchmarkLogger 5.4 µs 6.3 µs -13.98%
❌ BenchmarkChainWithSingleflight 3.1 µs 3.5 µs -11.38%
❌ BenchmarkChainRewritePassthrough 2.3 µs 2.6 µs -11.36%
❌ BenchmarkChainStaticFile304 2.7 µs 3 µs -10.76%
⚡ idle 4 ns 3 ns +33.33%
⚡ BenchmarkInternH2HeaderName 61 ns 49 ns +24.49%
⚡ 4producers 169 ns 139 ns +21.58%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing fix/celeris-732-720-retained-views (30c5c41) with main (dccb839)2

Open in CodSpeed

Footnotes

  1. 5 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on main (a64f920) during the generation of this report, so dccb839 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api Public-facing API surface area/observability Metrics, logging, debug endpoints bug Something isn't working middleware Middleware implementation security Security hardening

Projects

None yet

1 participant