Worker: Runs get lost if the dataclip is >150mb

[josephjclark]

On Staging, with a 200mb dataclip, I'm seeing this pattern of behaviour

• The step complete with pretty high memory usage: [R/T] ❯ Final memory usage: [step 515mb] [system 603mb]
• The step complete message is sent to lightning
• The run complete message seems to fail to send
• Maybe 1 second later, the server seems to restart. No exception or message in the log.

Since run:complete fails to send, the run will be Lost

Here's an example on staging

Here's logs on GCP