First step of implementing CEP-002: Change container structure

[maxnoe]

• Closes Discussion on restructuring Event Container structure #1165
• Closes Rename *Camera*Containers to *Telescope*Containers #2170
• Closes Should trigger / pointing information be stored into in dl0? #2374
• Closes R1/DL0 telescope event type is also stored in trigger. #2952

[maxnoe]

Before I continue changing everything, could you review the new container structure?

I think it's already a big step in the right direction, as the structure is much simpler, we could get rid of many indirections...

[nbiederbeck]

This comment just looked at ctapipe/containers.py. In general I think the structure looks fine, it makes more sense as seen in the tests, but while we're at it, some comments:

• For the EventContainers: and why do we have count?

• Should there be a general MuonContainer?

• Why does the TelescopePointingContainer have no ra/dec?
  Or why does ArrayPointingContainer have it?

• Should we rename the ParticleClassificationContainer and TelescopeImpactParameterContainer to be more consistent with ReconstructedEnergyContainer and ReconstructedGeometryContainer? E.g. ReconstructedParticleContainer, ReconstructedImpactContainer.

• Is Reconstructed[…].telescopes sensible to be None if Mono? Or should it just hold the corresponding tel_id and be of type int?

• Is TelescopeTriggerContainer.n_trigger_pixels with default -1 sensible? Or is 0 better to have an unsigned int?

• SimulationConfigContainer have many nans, but str seems more appropriate given the description. Or None.

• TelescopeSimulationContainer.impact is not called true_impact.

• MorphologyContainer, is -1 a better default than 0? Does it make a difference of default 0 and filled 0?

• Is CameraTimingParametersContainer still needed?

• LeakageContainer.intensity_width_{1,2} is ambiguously documented (intensity vs. fraction of intensity, as of the docstring).

• There are some Camera-containers, which I think we can get rid of?

[maxnoe]

Why does the TelescopePointingContainer have no ra/dec?

Because all our telescopes are alt/az mounts, so this is the actual information that is monitored.

Or why does ArrayPointingContainer have it?

Because that's the reference point the array is tracking, which might be in sky-fixed coordinates.

TelescopeSimulationContainer.impact is not called true_impact.

Isn't this accomplished by adding the true_ prefix in the data writer?

Is TelescopeTriggerContainer.n_trigger_pixels with default -1 sensible? Or is 0 better to have an unsigned int?

I think it's important to distinguish invalid from valid data, 0 trigger pixels are possible (e.g. for pedestal events which are randomly triggered), so better distinguish. Same for the other positive integers.

[maxnoe]

For the EventContainers: and why do we have count?

Just nice to have I guess.

[maxnoe]

Is CameraTimingParametersContainer still needed?

That's connected to all the other Camera... containers and to #2141, we need them as long as we still support computing image parameters in CameraFrame

[StFroese]

The ArrayEventContainer contains the DL2 data (geometry, energy, classification) of the event in the ArrayDL2Container.
It also contains the telescopes as a map of TelescopeEventContainer's which contains all data levels like r0... and also dl2. dl2 in this case is a TelescopeDL2Container which is a subclass of the ArrayDL2Container again.

Why do we save the array event reconstructions at telescope level again? This is a little bit confusing to me

[maxnoe]

Why do we save the array event reconstructions at telescope level again? This is a little bit confusing to me

The DL2TelescopeContainer is for monoscopic predictions, not for repeating the subarray info.

The inheritance here is just to get the same fields, but I agree that it's confusing that isinstance(DL2TelescopeContainer, DL2SubarrayContainer). I had an intermediate version where both just inherited from Container and then duplicated the field definitions.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[StFroese]

The inheritance here is just to get the same fields, but I agree that it's confusing that isinstance(DL2TelescopeContainer, DL2SubarrayContainer). I had an intermediate version where both just inherited from Container and then duplicated the field definitions.

Can we add a ReconstrucedShowerContainer which contains the fields and DL2{Telescope|Subarray}Container inherit from that?

[maxnoe]

Hi all,

I rebased this on the current main and squashed the commits into one, to ease follow-up rebases when we make updates to main.

I'd appreciate a fresh review now.

I suggest to first only review the container structure in ctapipe.containers (beware, this is collapsed by default) before reviewing any of the resulting code changes.

Once we are happy with the container structure and renaming, I'll fix up the rest of the code and we can commence the review there.

[ccossou]

For the EventContainers: and why do we have count?

Just nice to have I guess.

On this part. This, to me, is related to how people parse the file and shouldn't be in the container itself, or if it is, should be hidden and not publicly exposed (even if in python, private just means harder to type). But that's a minor comment and not the main purpose of CEP-002.

[ccossou]

(Sorry, commented on file without realizing it wasn't container.py anymore, got carried away.)

[maxnoe]

@kosack @mexanick This is now rebased on current main (not fun!) and locally tests are passing again, but I see a lot of warnings I didn't expect, so I'll continue tomorrow.

In any case, I would be very helpful if you could already start reviewing the changes in the container definitions src/ctapipe/containers.py

and maybe play around a bit interactively with the new structure.

[maxnoe]

There is one change in the file format here as well, not just to the in-memory containers: to mirror the move of trigger information to dl0, this also addresses #2374 (and #2952) which better conforms with the CTAO data model.

This however should be implemented in a backwards-compatible way (reading old files with these containers stored in /dl1 is still supported).

[maxnoe]

Add migration guide details here.

[mexanick]

I think the DL1 trigger table paths should be removed as that tables do not exist anymore

[maxnoe]

I think the DL1 trigger table paths should be removed as that tables do not exist anymore

These are crucial for backwards compatibility with older data and are used in e.g. the HDF5EventSource and TableLoader.

We do not remove compatibility with older data formats in this pull request.

[ctao-sonarqube]

Quality Gate passed

Issues
0 New issues
6 Fixed issues
4 Accepted issues

Measures
0 Security Hotspots
95.8% Coverage on New Code
0.4% Duplication on New Code

See analysis details on SonarQube

[kosack]

EDIT: never mind, I was looking at RandomForestRegressor_energy and not RandomForestRegressor_tel_energy

I think there is an iteration bug: I loaded an events file with TableLoader and with EventSource, and compared the first event's energy and get different results for the telescope-wise energy:

[kosack]

Looks very good! i don't see any major issues, and anything else can be cleaned up in future PRs. I guess you still want to add a migration guide, but I can re-approve once that is done.

[kosack]

Note that I opened #2997 to fix a long-standing bug, which made exploring the new structure a bit less painful interactively.

[maxnoe]

FYI: here is a PR in the ctapipe_io_zfits plugin working with this branch: https://github.com/cta-observatory/ctapipe_io_zfits/pull/22/changes