The reading code in extra.components tries to work around HDF problems with respect so slicing and thus reads entire datasets into memory for subsequent slicing. This relies on virtual memory tricks allowing to allocate potentially very large amounts of memory, even overcomitting physical memory.
This has shown to fail on shared nodes under high memory pressure, potentially due to different settings. Selecting down a detector object to a single train and pulse still caused MultimodKeyData.ndarray() to allocate an entire sequence worth of memory, which failed.
The reading code in
extra.componentstries to work around HDF problems with respect so slicing and thus reads entire datasets into memory for subsequent slicing. This relies on virtual memory tricks allowing to allocate potentially very large amounts of memory, even overcomitting physical memory.This has shown to fail on shared nodes under high memory pressure, potentially due to different settings. Selecting down a detector object to a single train and pulse still caused
MultimodKeyData.ndarray()to allocate an entire sequence worth of memory, which failed.