gRPC timeout fix

[Saaketh0]

We got the error pasted below, which was due to the connection between them sitting idle for too long causing AWS to drop that connection.

The temp fix involved sending a request, waiting for it to time out and then for a new connection to be made, and then sending another request, reinstantiating all the connections like that.

The fix in this repo is broader, initiating options for the server and client to send pings every 30 seconds to each other when inactive, bypassing this inactive error.

Other errors that have surfaced from this are unbounded wait times that cause the user to not get a valid error in a fast enough time period, making an issue about that right now and will patch it up soon.

ERROR:__main__:Failed to send result callback to {Server IP}:50051: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "Stream removed (recvmsg:Connection timed out (110))" debug_error_string = "UNAVAILABLE:Stream removed (recvmsg:Connection timed out (110))"

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 63a5964a-953f-4a93-851e-da01aca0062d

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[nirajlcc]

Looks good for the dynamic connections between global and local controller. However please check for other connections like towards DB, between UI and Local controller.

[Saaketh0]

CAN-203

[Saaketh0]

Currently, when connecting via HTTP to port 8080, if the connecting connection got stalled for some reason and didn't drop the connection, no other connection was able to connect. Implemented threads for the connection so multiple could happen at the same time, as well as timeouts to automatically disconnect stale connections.

Will look at increased logging for these errors later.

Port 6379 connections were not an issue, as Redis manages them and has automatic retries + healing there.

[nirajlcc]

thanks, how did you test that the connection was stalled in reference to => Currently, when connecting via HTTP to port 8080, if the connecting connection got stalled for some reason and didn't drop the connection, no other connection was able to connect. Implemented threads for the connection so multiple could happen at the same time, as well as timeouts to automatically disconnect stale connections.

[Saaketh0]

Mentioned in standup but also leaving another comment here for observability, I didn't. I went looking through the connections for all the other ports, and for this connection saw that there was no error handling/timeout logic. Worst case, this fix does absolutely nothing but this could also prevent this connection from getting jammed.