Distributor: return HTTP 499 instead of 500 on client push cancellation#7717
Open
AllenYTF wants to merge 2 commits into
Open
Distributor: return HTTP 499 instead of 500 on client push cancellation#7717AllenYTF wants to merge 2 commits into
AllenYTF wants to merge 2 commits into
Conversation
Maps client-canceled remote-write push requests to HTTP 499 (Client Closed Request) instead of 500, so ordinary client-side cancellations (disconnects, or a fronting proxy/LB timeout) are no longer counted as server-side errors. ring.DoBatch returns a bare context.Canceled when the client cancels the request context. Unlike the other error paths in the distributor's Push (which wrap via httpgrpc.Errorf), this error carries no gRPC status, so httpgrpc.HTTPResponseFromError returns ok == false and the handler falls through to the generic 500. The query-frontend already handles this correctly (StatusClientClosedRequest = 499); this ports the same convention to the write path. Changes: - pkg/util/push/push.go: in the PRW1 and PRW2 handlers, map context.Canceled (via errors.Is, so wrapped cancellations are also caught) to httpgrpc.Errorf(util_api.StatusClientClosedRequest, ...) before status extraction, and exclude 499 from the push refused warn log. - pkg/util/push/otlp.go: same mapping for the OTLP write path. - Tests for bare and wrapped cancellation across the PRW1, PRW2, and OTLP handlers. Signed-off-by: Allen Yan <allen_yan@apple.com>
Signed-off-by: Allen Yan <allen_yan@apple.com>
AllenYTF
force-pushed
the
return-499-on-client-push-cancellation
branch
from
July 23, 2026 01:47
8b50c56 to
c9509d8
Compare
krunaljain
approved these changes
Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Maps client-canceled remote-write and OTLP push requests to HTTP
499(Client Closed Request) instead of500, so ordinary client-side cancellations (disconnects, or a fronting proxy/LB timeout) are no longer counted as server-side errors.Fixes #7713.
Root cause
ring.DoBatchreturns a barecontext.Canceled(pkg/ring/batch.go,return ctx.Err()) when the client cancels the request context. Unlike the other error paths in the distributor'sPush(which wrap viahttpgrpc.Errorf), this error carries no gRPC status, so in the push HTTP handlerhttpgrpc.HTTPResponseFromErrorreturnsok == falseand the handler falls through to the generichttp.StatusInternalServerError.The query-frontend already handles this correctly (
pkg/frontend/transport/handler.go,StatusClientClosedRequest = 499). This ports the same convention to the write path.Changes
pkg/util/push/push.go: in the PRW1 and PRW2 handlers, mapcontext.Canceled(viaerrors.Is, so wrapped cancellations are also caught) tohttpgrpc.Errorf(util_api.StatusClientClosedRequest, ...)before status extraction, reusing the sharedStatusClientClosedRequest = 499constant. Excludes499from the "push refused" warn log.pkg/util/push/otlp.go: same mapping for the OTLP write path.Impact
Client-canceled push requests are currently logged and alerted on as 5xx errors, which can trigger false-positive SLO/error-rate alerts and pollute error dashboards. After this change they return
499.🤖 AI assistance disclosure: This change was prepared with the help of Claude Code. The author has reviewed and understands every line and takes full responsibility per the project's GenAI policy.