Skip to content

(Potential) memory leak with uproot.iterate #1528

Closed

Description

@pfackeldey

(this is tested in uproot v5.6.5, likely present in other versions as well).

The issue & reproducer

When running the following snippet and benchmarking this with memray:

import uproot

def loop_iterate(rootfile):
    with uproot.open(rootfile, array_cache=None, object_cache=None) as f:
        tree = f["tree"]
        for batch in tree.iterate(step_size="200 MB", library="ak"):
            print(repr(batch))


if __name__ == "__main__":
    loop_iterate("~/Downloads/zlib9-jagged0.root")

I'm getting a memory consumption of the physical RAM usage (RSS) of up to 1.6GB (even though the step size is 200 MB):

Image

This is already surprising, however another indication that there's something up is that an explicit gc.collect at the end of each iteration improves the RSS situation by ~2x, i.e.:

import uproot
+import gc

def loop_iterate(rootfile):
    with uproot.open(rootfile, array_cache=None, object_cache=None) as f:
        tree = f["tree"]
        for batch in tree.iterate(step_size="200 MB", library="ak"):
            print(repr(batch))
+           del batch
+           gc.collect()


if __name__ == "__main__":
    loop_iterate("~/Downloads/zlib9-jagged0.root")

which gives up to 800MB RSS consumption:

Image

Why is this bad?

RSS is the physical RAM usage by this process, which dask monitors to decide if a worker should be killed due to OOM or not.

What I've found so far...

The memory usage grows by the following function: https://github.com/scikit-hep/uproot5/blob/main/src/uproot/behaviors/TBranch.py#L1440-L1452 and to be more explicit by this part of it: https://github.com/scikit-hep/uproot5/blob/main/src/uproot/behaviors/TBranch.py#L3421-L3428

What does work correctly is that the file arrays dictionary by the above function is ~200 MB, that's good! However, this _ranges_or_baskets_to_arrays still uses ~800 MB to fill the ~200 MB arrays dict and does not free it again.

Also, the "popper-trick" that @jpivarski introduced in #1305 does enable the manual gc.collect to help here (without it even that won't help).

So, my understanding right now is that uproot.iterate does yield correctly sized arrays, but it uses way to much memory while doing so and also doesn't free it properly.

Other implications

_ranges_or_baskets_to_arrays is used also in other loading functions, and some quick tests showed that:

def loop_manual(rootfile):
    with uproot.open(rootfile, array_cache=None, object_cache=None) as f:
        tree = f["tree"]

        starts = list(range(0, tree.num_entries, tree.num_entries // 20))
        stops = starts[1:] + [tree.num_entries]
        ranges = list(zip(starts, stops))
        for start, stop in ranges:
            print(f"entry {start} to {stop}")
            entry = tree.arrays(entry_start=start, entry_stop=stop, library="ak")
            print(repr(entry))


def loop_same_chunks(rootfile):
    with uproot.open(rootfile, array_cache=None, object_cache=None) as f:
        tree = f["tree"]

        chunk_starts = 0
        chunk_stops = 53687091
        for _ in range(10):
            print(f"entry {chunk_starts} to {chunk_stops}")
            entry = tree.arrays(entry_start=chunk_starts, entry_stop=chunk_stops, library="ak")
            print(repr(entry))

have a similar memory behavior, see e.g. the profile for loop_manual (the numerical values of the y axis is different of course because I can't exactly mirror "200 MB" steps by hand):

Image

and for loop_same_chunks:

Image

What I want to see / was expecting

The orange and blue line overlap and roughly follow a saw tooth shape with 200 MB jumps per iteration (and not much additional overhead in RAM).


This has been originally been found by @oshadura in the scope of the integration challenge, here I just attach a local reproducer with some first findings.

cc @oshadura @alexander-held

Activity

  1. added
    bug (unverified)The problem described would be a bug, but needs to be triaged
    on Nov 28, 2025
  2. ianna commented on Nov 28, 2025

    @ianna
    Member

    @pfackeldey - thanks for the detailed report! Indeed, it is a serious issue. I will have a look asap. Thanks!

  3. pfackeldey commented on Nov 28, 2025

    @pfackeldey
    CollaboratorAuthor

    Thanks @ianna, I'll try debugging as well meanwhile.
    I suspect this is another ref-cycle (or similar) issue like @jpivarski and me solved in #1305 which prevents this memory from being garbage collected. Those tend to be hard to find from my previous debugging experience 😅

  4. ikrommyd commented on Nov 28, 2025

    @ikrommyd
    Member

    Good find @pfackeldey. I'm wondering, what is your interpretation for the fact that resident size flattens out after some iterations? It's like it "stops leaking" after a bit.

  5. pfackeldey commented on Dec 2, 2025

    @pfackeldey
    CollaboratorAuthor

    @ikrommyd I do not know why it looks like this

  6. pfackeldey commented on Dec 2, 2025

    @pfackeldey
    CollaboratorAuthor

    I checked 4 'versions' of {TTree,RNTuple}.iterate in addition with an open data NanoAOD file (~1GB decompressed in total, stepsize="20 MB").

    1. TTree
    2. TTree + explicit gc.collect() at the end of every loop iteration
    3. RNTuple
    4. RNTuple + explicit gc.collect() at the end of every loop iteration

    The results are as follows:

    TTree

    Image

    TTree + explicit gc.collect() at the end of every loop iteration

    Image

    RNTuple

    Image

    RNTuple + explicit gc.collect() at the end of every loop iteration

    Image

    Summary

    In the TTree case an explicit gc.collect() does help apparently. RSS is piling up in every iteration and it looks to me like the yielded 20 MB array is cleaned up properly (the little peaks) but somehow RAM is still getting full.

    In the RNTuple case I suspect that the whole array is read first and then chunks of that are yielded. Although that would only explain it growing to 1GB. Probably there are some copies?

    In general I think it is expected that the memory consumption is somewhat larger than 20 MB because uproot will have the compressed and decompressed chunk in memory at the same time. So in the worst case it should be 2x the chunk size, so 40 MB in this case... but not 1.4GB in TTree, or 2.5GB in RNTuple.

    @ariostas any ideas why RNTuple looks like this?

    I'm honestly pretty stuck here, and after days of debugging I'm not sure where else to look into. Any ideas? (maybe also @nsmith- ?)

    (fyi: changing the stepsize to an int for the number of events doesn't change anything)


    TTree code

    with uproot.open(TTREE_FILE, array_cache=None, object_cache=None) as f:
      tree = f["Events"]
      for i, batch in enumerate(tree.iterate(step_size="20 MB", library="ak")):
        print(repr(batch))
        # uncomment to enable explicit gc.collect()
        # del batch
        # gc.collect()

    RNTuple code

    with uproot.open(RNTUPLE_FILE, array_cache=None, object_cache=None) as f:
      rntuple = f["ntuple"]
      for i, batch in enumerate(rntuple.iterate(step_size="20 MB", library="ak")):
        print(repr(batch))
        # uncomment to enable explicit gc.collect()
        # del batch
        # gc.collect()
  7. pfackeldey commented on Dec 2, 2025

    @pfackeldey
    CollaboratorAuthor

    Ok, so I found apparently the main reason (and some solutions) to this issue.

    Apparently, we're suffering in uproot from memory fragmentation of arenas created by malloc. When I switch the allocator to jemalloc (which has a much tighter fragmentation behavior) on MacOS it looks way better, see:

    Image

    The solutions to this are basically:

    1. glibc only: adjust malloc options through env vars (e.g. MALLOC_ARENA_MAX or MALLOC_TRIM_THRESHOLD)
    2. glibc only: call ctypes.CDLL("libc.so.6").malloc_trim(0) after yielding batches
    3. use a different allocator with tighter fragmentation behavior (e.g. jemalloc)
    4. avoid allocating as much as possible in uproot, i.e. can we reuse buffers e.g. for decompressing?
    5. (write uproot in a low level language to have more control over allocations?)

    I didn't test 1-2 because I don't have a linux machine, but that should help.

    (Also this here should help for dask: https://distributed.dask.org/en/stable/worker-memory.html#memory-not-released-back-to-the-os)

  8. nsmith- commented on Dec 2, 2025

    @nsmith-
    Member

    Memory fragmentation has been a problem for us since ~forever, unfortunately. When we have many small long-lived allocations interspersed with very large array allocations, the arenas end up mostly empty. See this old coffea issue e.g. scikit-hep/coffea#249

  9. pfackeldey commented on Dec 2, 2025

    @pfackeldey
    CollaboratorAuthor

    Memory fragmentation has been a problem for us since ~forever, unfortunately. When we have many small long-lived allocations interspersed with very large array allocations, the arenas end up mostly empty. See this old coffea issue e.g. scikit-hep/coffea#249

    yes, that makes sense. We did quite some work in awkward to reduce small long-lived allocations, so that should hopefully be noticeable already. Do you think we could reuse some buffers explicitly in uproot @nsmith- (or do you have any other idea how that could be mitigated apart from "uproot-rs" :D)?

    This issue is not a programmatic memory leak then, should we turn this into a discussion instead for future reference?

  10. bfis commented on Dec 5, 2025

    @bfis

    For glibc malloc, the fragmentation can also be limited by setting M_MMAP_THRESHOLD to small-ish values (e.g. 1024), such that most of the ephemeral allocations during iteration get served via mmap. However, this is likely to cause majors slowdowns, up to orders of magnitude.

    Additionally, it may be that the fragmentation in uproot is exacerbated by the dynamic adjustment of this mmap threshold, which is supposedly enabled by default.

  11. locked and limited conversation to collaborators on Dec 9, 2025
  12. converted this issue into a discussion #1535 on Dec 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug (unverified)The problem described would be a bug, but needs to be triaged

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions