Reading a std::vector<struct> inside another std::vector<struct>

[asnaylor]

Hi,
I'm trying to read variables inside a vector of structs which is inside a vector of structs, here's the simplified c++ header of the ROOT file:

struct stepInfo
{
    Int_t StepNumber;
    std::string VolumeName;
    Double_t Position_mm[3];
    Double_t ParticleEnergy_keV;
};
struct trackInfo
{
    std::string ParticleName;
    Int_t TrackID;
    std::vector<stepInfo> steps;
};

std::vector<trackInfo> tracks; // TBranch

Uproot can load tracks.ParticleName into an Awkward Array fine but when you try to load tracks.steps you get:

NotImplementedError: memberwise serialization of AsArray

I have seen there is an open issue #38 about ROOT memberwise splitting but i wondered if i can force the interpretation, tell uproot what the structure was e.g.:

interpretation = uproot.AsDtype(....)
array = uproot_tree['tracks.steps'].array(interpretation)

Is this possible? If so how do you actually go about doing this?

Thanks

[jpivarski]

As is intended by the NotImplementedError message, this is an unsolved problem. It's not just a matter of Uproot failing to recognize the interpretation, in which case I could give you a hand-constructed Interpretation that you could apply. It's that we don't have any algorithms written in Python that will deserialize data in that form. (Also, it's not just the fact that two structs and a std::vector are nested: ROOT can serialize such objects without the "memberwise" option being turned on, and Uproot can read that data. It's this "memberwise" switch that we haven't figured out.)

Here's the central issue on the problem: #38

Here is a first step, in which @kratsg has implemented part of it: #298

And here are two unfinished drafts to go further with it: #314 and #318.

If you want to look into this, you can see the raw bytes in one entry (entry i) by:

uproot_tree['tracks.steps'].debug(i)
uproot_tree['tracks.steps'].debug_array(i)

They have docstrings that are also available online here and here. The first is for printing out the bytes in a way that may be helpful for debugging, and the second returns the data as an uninterpreted NumPy array for programmatic access.

From what you say about being able to read particleName, it looks like the TBranches are split, so the TBranch tracks.steps would only be the step values. The general pattern with memberwise splitting is that it would put all the "StepNumber" values first, then all the "VolumeName", then all the "Position_mm", then all the "ParticleEnergy_keV" (whereas non-memberwise would have one "StepNumber", followed by one "VolumeName", one "Position_mm", and one "ParticleEnergy_keV" before moving on to the next "StepNumber", etc.). The std::vector would also have some header to indicate how many stepInfos there are. Everything is big-endian, so if you're interpreting them with Python's struct.unpack, a > would be needed in the format string.

That's all the information that I can give you right now. Also, if you post your ROOT file here, someone might be able to help you.

[asnaylor]

Thanks for the quick response. Ah right, so the way/order the variables are stored is different. My experiment has moved onto a more uproot friendly ROOT format, it's just the old files that are like this so I'll just use pyROOT to pull out the data of those files.

[jpivarski]

That's a great idea, switching to PyROOT for the older files, as that certainly works. The development effort needed for this would make a lot more sense if it were addressing a new or upcoming data format.

There seems to be a trend now toward simpler data. CMS's NanoAOD is getting a lot of attention, and ATLAS has had a few similar projects going for a while now. It's interesting to see particles measured in keV getting the same treatment!

[kratsg]

Yes, like @jpivarski said, this is how it works. I think we learned this clearest from the std::pair example, because we learned that the first thing you need to do is make a "shell" of all the various objects in the event, and then read each member one by one. So the two different ways are:

non-memberwise

[[Jet0.X, Jet0.Y, Jet0.Z, Jet0.M, ...], [Jet1], ... [JetN]]

versus memberwise

[[Jet0.X, Jet1.X, Jet2.X, ..., JetN.X], [Jet0.Y, ..., JetN,Y], [Jet.Z], ....]

You can almost think of it as the transpose in a way.