Fix canceled distributor push requests as 499 instead of 500

[wgliang]

Signed-off-by: wangguoliang iamwgliang@gmail.com

What this PR does:
FIx canceled distributor push requests as 499 instead of 500.

FYI：https://www.webfx.com/web-development/glossary/http-status-codes/

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yeya24]

LGTM

[wgliang]

All ready to merge.

[damnever]

This change may cause users to lose data since Prometheus(agent) does not retry on 499.

[friedrichg]

@damnever The 499 is never returned to grafana agent. The user canceled the request so they won't get the reply.
This is similar to #2156

[friedrichg]

I bet many users are going to understand this line as clients will receive 499. Can you make it obvious that this only affects internal metrics and logs?

[damnever]

@friedrichg IMHO, context could be canceled by the distributor level timeout(-distributor.remote-timeout), so the clients may still receive the 499 response.

[friedrichg]

@damnever you are absolutely right. We should continue to return 500 if the remote-timeout is hit. This PR should ensure that. Good catch

[damnever]

The context.Canceled isn’t always triggered by the root context, the cause could be any context inside the push, there are dependencies, and so on. We need to distinguish the difference. When the context.Canceled is returned in the middle of pushing data to ingesters (maybe this one?), then we may lose data since the Prometheus agent doesn’t retry on 499.

[wgliang]

I will close this PR and re-issue a fix if there is a better solution. Thank you for your feedback. @damnever @friedrichg