Worker: Runs get lost if the dataclip is >150mb

[josephjclark]

On Staging, with a 200mb dataclip, I'm seeing this pattern of behaviour

• The step complete with pretty high memory usage: [R/T] ❯ Final memory usage: [step 515mb] [system 603mb]
• The step complete message is sent to lightning
• The run complete message seems to fail to send
• Maybe 1 second later, the server seems to restart. No exception or message in the log.

Since run:complete fails to send, the run will be Lost

Here's an example on staging

Here's logs on GCP

[josephjclark]

Need to try and repro this again since recent changes. I suspect that the run thread will now be OOMKilled when trying to size these large objects.

I'd still like to work out a solution to #926 which would make this a lot more stable

[josephjclark]

I think we're seeing this again recently: large dataclips causing kubernetes to OOMkill the worker

here's a history where runs are regularly OOMkilled because they consume a lot of memory (this is good)

But every now and again one is lost. We see large memory spikes in grafana which sometimes lead to the worker being shut down.

What I'm sure is happening is that the worker thread is chewing up more memory than it's allowed to, and kubernetes is then killing it before it can fail itself. That's the json.stringify call, which is synchronous.

Some notes:

I also note that in that workflow, one step that runs just before a blow up has no code. No user action and STILL the OOMKill - presumably that's just the cost of processing the JSON dataclip inside the runtime.

app link

[josephjclark]

I'm pretty sure this is as simple as the JSON.stringify call which runs at the end of every step. Last time I looked at this in #926 I didn't come up with a satisfactory solution.

But wait! What we did over in 926 was implement a streaming algorithm to calculate the object size in the engine. But a few notes on this:

1. The runtime doesn't use the streaming approach to serialize the object at the end of the step. We only looked at the size-of-a-payload calculation, not the internal serializer (right?)
2. Looking at the code in the engine it looks like we're actually using the stringify algorithm, not the streaming one. So did we turn this off? Was this just a really dumb error?

[josephjclark]

Ok, the runtime does use steaming at the end of each step so that should be stable.

With a bit of luck we can just switch to the streaming algo in the engine and that will help.