Repository navigation
Description
(this is tested in uproot v5.6.5, likely present in other versions as well).
The issue & reproducer
When running the following snippet and benchmarking this with memray:
import uproot
def loop_iterate(rootfile):
with uproot.open(rootfile, array_cache=None, object_cache=None) as f:
tree = f["tree"]
for batch in tree.iterate(step_size="200 MB", library="ak"):
print(repr(batch))
if __name__ == "__main__":
loop_iterate("~/Downloads/zlib9-jagged0.root")I'm getting a memory consumption of the physical RAM usage (RSS) of up to 1.6GB (even though the step size is 200 MB):
This is already surprising, however another indication that there's something up is that an explicit gc.collect at the end of each iteration improves the RSS situation by ~2x, i.e.:
import uproot
+import gc
def loop_iterate(rootfile):
with uproot.open(rootfile, array_cache=None, object_cache=None) as f:
tree = f["tree"]
for batch in tree.iterate(step_size="200 MB", library="ak"):
print(repr(batch))
+ del batch
+ gc.collect()
if __name__ == "__main__":
loop_iterate("~/Downloads/zlib9-jagged0.root")which gives up to 800MB RSS consumption:
Why is this bad?
RSS is the physical RAM usage by this process, which dask monitors to decide if a worker should be killed due to OOM or not.
What I've found so far...
The memory usage grows by the following function: https://github.com/scikit-hep/uproot5/blob/main/src/uproot/behaviors/TBranch.py#L1440-L1452 and to be more explicit by this part of it: https://github.com/scikit-hep/uproot5/blob/main/src/uproot/behaviors/TBranch.py#L3421-L3428
What does work correctly is that the file arrays dictionary by the above function is ~200 MB, that's good! However, this _ranges_or_baskets_to_arrays still uses ~800 MB to fill the ~200 MB arrays dict and does not free it again.
Also, the "popper-trick" that @jpivarski introduced in #1305 does enable the manual gc.collect to help here (without it even that won't help).
So, my understanding right now is that uproot.iterate does yield correctly sized arrays, but it uses way to much memory while doing so and also doesn't free it properly.
Other implications
_ranges_or_baskets_to_arrays is used also in other loading functions, and some quick tests showed that:
def loop_manual(rootfile):
with uproot.open(rootfile, array_cache=None, object_cache=None) as f:
tree = f["tree"]
starts = list(range(0, tree.num_entries, tree.num_entries // 20))
stops = starts[1:] + [tree.num_entries]
ranges = list(zip(starts, stops))
for start, stop in ranges:
print(f"entry {start} to {stop}")
entry = tree.arrays(entry_start=start, entry_stop=stop, library="ak")
print(repr(entry))
def loop_same_chunks(rootfile):
with uproot.open(rootfile, array_cache=None, object_cache=None) as f:
tree = f["tree"]
chunk_starts = 0
chunk_stops = 53687091
for _ in range(10):
print(f"entry {chunk_starts} to {chunk_stops}")
entry = tree.arrays(entry_start=chunk_starts, entry_stop=chunk_stops, library="ak")
print(repr(entry))have a similar memory behavior, see e.g. the profile for loop_manual (the numerical values of the y axis is different of course because I can't exactly mirror "200 MB" steps by hand):
and for loop_same_chunks:
What I want to see / was expecting
The orange and blue line overlap and roughly follow a saw tooth shape with 200 MB jumps per iteration (and not much additional overhead in RAM).
This has been originally been found by @oshadura in the scope of the integration challenge, here I just attach a local reproducer with some first findings.
Activity
ianna commentedon Nov 28, 2025
@pfackeldey - thanks for the detailed report! Indeed, it is a serious issue. I will have a look asap. Thanks!
pfackeldey commentedon Nov 28, 2025
Thanks @ianna, I'll try debugging as well meanwhile.
I suspect this is another ref-cycle (or similar) issue like @jpivarski and me solved in #1305 which prevents this memory from being garbage collected. Those tend to be hard to find from my previous debugging experience 😅
ikrommyd commentedon Nov 28, 2025
Good find @pfackeldey. I'm wondering, what is your interpretation for the fact that resident size flattens out after some iterations? It's like it "stops leaking" after a bit.
pfackeldey commentedon Dec 2, 2025
@ikrommyd I do not know why it looks like this
pfackeldey commentedon Dec 2, 2025
I checked 4 'versions' of
{TTree,RNTuple}.iteratein addition with an open data NanoAOD file (~1GB decompressed in total, stepsize="20 MB").gc.collect()at the end of every loop iterationgc.collect()at the end of every loop iterationThe results are as follows:
TTree
TTree + explicit
gc.collect()at the end of every loop iterationRNTuple
RNTuple + explicit
gc.collect()at the end of every loop iterationSummary
In the TTree case an explicit
gc.collect()does help apparently. RSS is piling up in every iteration and it looks to me like the yielded 20 MB array is cleaned up properly (the little peaks) but somehow RAM is still getting full.In the RNTuple case I suspect that the whole array is read first and then chunks of that are yielded. Although that would only explain it growing to 1GB. Probably there are some copies?
In general I think it is expected that the memory consumption is somewhat larger than 20 MB because uproot will have the compressed and decompressed chunk in memory at the same time. So in the worst case it should be 2x the chunk size, so 40 MB in this case... but not 1.4GB in TTree, or 2.5GB in RNTuple.
@ariostas any ideas why RNTuple looks like this?
I'm honestly pretty stuck here, and after days of debugging I'm not sure where else to look into. Any ideas? (maybe also @nsmith- ?)
(fyi: changing the stepsize to an int for the number of events doesn't change anything)
TTree code
RNTuple code
pfackeldey commentedon Dec 2, 2025
Ok, so I found apparently the main reason (and some solutions) to this issue.
Apparently, we're suffering in uproot from memory fragmentation of arenas created by
malloc. When I switch the allocator tojemalloc(which has a much tighter fragmentation behavior) on MacOS it looks way better, see:The solutions to this are basically:
MALLOC_ARENA_MAXorMALLOC_TRIM_THRESHOLD)ctypes.CDLL("libc.so.6").malloc_trim(0)after yielding batchesjemalloc)I didn't test 1-2 because I don't have a linux machine, but that should help.
(Also this here should help for dask: https://distributed.dask.org/en/stable/worker-memory.html#memory-not-released-back-to-the-os)
nsmith- commentedon Dec 2, 2025
Memory fragmentation has been a problem for us since ~forever, unfortunately. When we have many small long-lived allocations interspersed with very large array allocations, the arenas end up mostly empty. See this old coffea issue e.g. scikit-hep/coffea#249
pfackeldey commentedon Dec 2, 2025
yes, that makes sense. We did quite some work in awkward to reduce small long-lived allocations, so that should hopefully be noticeable already. Do you think we could reuse some buffers explicitly in uproot @nsmith- (or do you have any other idea how that could be mitigated apart from "uproot-rs" :D)?
This issue is not a programmatic memory leak then, should we turn this into a discussion instead for future reference?
bfis commentedon Dec 5, 2025
For glibc malloc, the fragmentation can also be limited by setting
M_MMAP_THRESHOLDto small-ish values (e.g. 1024), such that most of the ephemeral allocations during iteration get served via mmap. However, this is likely to cause majors slowdowns, up to orders of magnitude.Additionally, it may be that the fragmentation in uproot is exacerbated by the dynamic adjustment of this mmap threshold, which is supposedly enabled by default.