resource eventlog grows without bound in memory

[garlick]

Problem: with #6586 merged, the memory used by the resource journal can grow without bound. This may be an issue on El Cap.

Before the next release we should think about how to mitigate that on large systems.

[grondo]

A simple way to address this issue would be to add a configurable maximum number of entries allowed in the in-memory resource journal.

To allow resource journal consumers to fully reconstruct state from a truncated journal, a new truncate or similar event could be added to RFC 44 which captures the state at the start of a truncated journal. This event would, at a minimum, contain the online and offline idsets at the time of truncation as well as the current drain state, possibly reusing the same format as the drain key in a resource.status response.

I'm not sure the entire state in the truncate event would immediately be necessary for the current use case, so the extra information could be marked as OPTIONAL for now. If and when it does become a requirement, it seems possible that the "truncate state" in the journal could be kept up to date by apply the newly truncated events to the previous "truncate state" as they are dropped.