Why

While working on the hydra-doom project, we noticed that the in memory state grew without bound (see #1572). This meant that, at the sustained load that the hydra doom demo was producing, nodes became inoperable after just a few hours.

One of the culprits was the ?history of the hydra-node websocket API. That feature sends clients the whole history by default (can be turned off, see documentation https://hydra.family/head-protocol/docs/api-behavior#replay-of-past-server-outputs)

We want to make Hydra heads be operable for long times even with sustained load, which requires us to bound any memory usage (and even disk usage, see #1581).

What

• The default behavior of the websocket API is to not send the whole history, but only the Greetings message.
  • This breaking change is to be communicated properly
• The API server history, if requested by ?history=true, is loaded on-demand from disk.

How

• Write an ADR describing changing the architecture of hydra-node such that the APIServer is an interpreter of StateEvent stream #1717
• Remove ServerOutput type and instead use StateChanged type #1800
• Measure RAM usage for hydra-node #1803
• Refactor the API Server
  • API-internal Projection read models need to aggregate over StateEvent
  • Re-use eventId as sequence number
  • Load events on startup and on-demand to initialize read models and next sequence number.
  • Load events on-demand if a new connection wants the full ?history
• Now the API server is just an EventSink
• Websocket needs to emit this information on new state changes