When we roll out new versions of the ingesters, they sometimes fail to correctly deregister themselves from consul, causing user-visible errors, manifesting as 500s on /push requests.
Distributor logs will have messages like:
time="2016-09-12T14:12:22Z" level=error msg="error sending request: Post http://10.244.9.10:80/push: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" source="server.go:107"
And the retrieval queues will likely fill up.
You can confirm this is a problem by looking at the distributor logs and observing:
time="2016-09-12T15:00:31Z" level=info msg="Got update to ring - 6 ingesters, 512 tokens" source="ring.go:85"
Where 6 is higher than the number of actual ingester pods.
The workaround is to restart consul like so:
kubectl delete --namespace frankenstein name=consul
After a minute or so, the errors in the distributor log should clear.
When we roll out new versions of the ingesters, they sometimes fail to correctly deregister themselves from consul, causing user-visible errors, manifesting as 500s on
/pushrequests.Distributor logs will have messages like:
And the retrieval queues will likely fill up.
You can confirm this is a problem by looking at the distributor logs and observing:
Where
6is higher than the number of actual ingester pods.The workaround is to restart consul like so:
After a minute or so, the errors in the distributor log should clear.