Asynchronous sender work thread terminated abnormally: Net::ReadTimeout

[dennis-tra]

Hey there,

just recently we started to experience the following error message:

ERROR -- application_insights: Asynchronous sender work thread terminated abnormally: Net::ReadTimeout

in conjunction with the fluentd plugin fluent-plugin-application-insights. This happens several times a day. Any idea how we could debug that issue?

Thanks in advance,
Dennis

[yantang-msft]

Looking at this exception: https://www.exceptionalcreatures.com/bestiary/Net/ReadTimeout.html
My gut feeling is AI's ingestion endpoint doesn't respond within some period of time.

[dennis-tra]

Hey Yanming Tang (@yantang-msft),

thanks for your fast reply.

Is there anything, that we can do to get to the root cause? Does this Ruby module have a retry mechanism? If so, is it guaranteed that we won't miss any logs if a second try is successful?

Cheers,
Denis

[yantang-msft]

Looking at the implementation, there is no retry logic for this sdk. While there is retry logic for the .Net sdk. You can try open a ticket at here: https://github.com/Microsoft/ApplicationInsights-Home for more suggestions.
There was a plan to have a common component written in .Net for logics like retry, sampling, so there is no need to replicate those logics for all language. I don't know how it is going right now, but it might be an option for you.

[dennis-tra]

Hey Sergey Kanzhelev (@SergeyKanzhelev),

I raised an Azure support ticket (119042426000987) and Sahil suggested to mention you here. I hope you can help us out.

Our logging setup consists of fluentd daemonsets that are using the https://github.com/Microsoft/fluent-plugin-application-insights fluentd plugin. This plugin in turn uses https://github.com/Microsoft/ApplicationInsights-Ruby. Some time we started to experience error messages in our fluentd logs. The problem happens across multiple application Insights resources. Here are some specific examples of error messages:

1. [2019-04-09T12:22:43.758543 #30387] ERROR -- application_insights: Asynchronous sender work thread terminated abnormally: Net::ReadTimeout
2. [2019-04-07T10:44:39.794258 #30387]  WARN -- application_insights: Failed to send data: Processing error
3. [2019-04-08T13:56:36.548045 #30387]  WARN -- application_insights: Failed to send data: Service Unavailable
4. [2019-04-09T10:11:56.923651 #30387]  WARN -- application_insights: Failed to send data: Internal Server Error
5. [2019-03-25T05:05:31.405414 #314] ERROR -- application_insights: Asynchronous sender work thread terminated abnormally: Connection reset by peer - SSL_connect
6. [2019-04-23T12:29:45.317970 #8445] ERROR -- application_insights: Asynchronous sender work thread terminated abnormally: Connection reset by peer

All of these errors happen multiple times. However, in between fluentd functions normally.

If such an error has occured, does that mean we are losing log messages or is there a retry mechanism?

[dennis-tra]

Hi Sergey Kanzhelev (@SergeyKanzhelev) I would really appreciate your input on this matter as the support just closed my ticket without further help:

[...], I see your ongoing issue on GitHub. Someone from Dev Team will definitely reply to it.