CellH5 memory usage #4329
Description
We occasionally see test failures in the cellh5 directory, unrelated to any open pull request, e.g. https://bf-testing-results.s3.amazonaws.com/2025/2025-05-06/cellh5.log
No stack trace or other informative logging is available when this happens.
I can pretty consistently reproduce locally with a low heap size:
$ ant -Dtestng.memory=64m -Dtestng.directory=~/data/bf-data-repo/automated-tests/curated/cellh5 -Dtestng.configDirectory=~/data_repo_config/curated/cellh5 test-automated
...
[testng] [2025-06-17 15:04:44,089] [main] Maximum heap size = 30 MB
[testng] Scanning for files...
[testng] [2025-06-17 15:04:44,801] [main] ----------------------------------------
[testng] [2025-06-17 15:04:44,802] [main] Total files: 11
[testng] [2025-06-17 15:04:44,802] [main] Scan time: 0.712 s (64 ms/file)
[testng] [2025-06-17 15:04:44,803] [main] ----------------------------------------
[testng] [TestNG] [Error]
[testng] The factory method class loci.tests.testng.FormatReaderTestFactory.createInstances() threw an exception
[testng] The tests failed.
Since this is a problem during the scanning phase, the thread count is irrelevant.
For now, I've tried moving the cellh5 directory from the start to the end of the batch on the test machine. That should mean less memory pressure once cellh5 is run, which hopefully helps. Longer term, I expect we'll need to look carefully at the CellH5Reader and JHDFServiceImpl to see if there is any way to better manage memory when calling jhdf.