Profile: Many Subsequent Python Runs #457
Description
In recent (external) sim optimization runs with @RTSandberg, we sometimes saw situations in which memory grew over time (leaked). We overcame them by ensuring AMReX init/finalize cycles per sim, with sim startup / closeout cycles were cleanly performed, including external model states we have in our programmable element. We also derived multiple programmable elements in this test to customize them.
While this works well, we still (sometimes/sporadically) see crashes, which could be from us or our external models in optimization. We currently have no time to debug them, thus this issue.
Nonetheless, a systematic test/benchmark to validate that running 100'000s of sims in a row does not create:
- invalid data accesses
- memory leaks
would be good to avoid that we overlook anything.