No option to write DL0 waveforms

[maxnoe]

Please describe the use case that requires this feature.

We have DataWriter options to write r0 and r1 waveforms, not for DL0.

Describe the solution you'd like

Also add option to write-out DL0.

[kosack]

Note that the difference from R1 is that DL0 waveforms are sparse, since they have DVR applied. That was the main reason they were not implemented in the past, but now it's clear we can just keep with full waveform arrays, but zero out the masked values and store a mask. It wont be as efficient as truly sparse storage with variable-length arrays, but storing DL0 is not a main use case of this format.

A few comments:

1. The dl0 waveform is supposed to be a masked array, according to the documentation. However, that is not easy to store, so I guess we should just store the full DL0 waveform array with zeroed out values, and provide a method that turns the pixel_status + DL0 waveforms back into a masked array.
2. For efficiency if R1 waveforms are stored, consider excluding the DL0 waveform (which would be redundant) to save some space. Not really required, but might be useful if we want to write out both R1 and DL0 for debugging purposes, you really only need the dl0.tel.pixel_status and not a second copy of the values (assuming there is no change in the values between R1 and DL0, which was the intention)
   .

[maxnoe]

Note that the difference from R1 is that DL0 waveforms are sparse, since they have DVR applied.

Sure, but some input files (most real data files) will only come with DL0 and it seems a bit weird to have support for everything else.

I just came across it when trying out #2780

[maxnoe]

The dl0 waveform is supposed to be a masked array, according to the documentation.

That's not the case at least for DL0 produced by our own sources and the Camera calibrator from R1. Would make sense though.

Not sure how that would propagate through the chain.

The mask itself is basically the pixel_status array, which would be written out along side the waveform when just writing the DL0 container.

[kosack]

That's not the case at least for DL0 produced by our own sources and the Camera calibrator from R1. Would make sense though.

That might make this easier then. I haven't looked at what is done for the LSTEventSource. I think the main point is that we don't yet well specify how to store and work on DVR'd DL0 data, so should discuss it when implementing this issue, and probably need to add some tests to ensure that DL1 algorithms work correctly with whatever method is used. The good thing is since it is not really being used in a standard way yet, we have the freedom to explore a few ways to do it.

[kosack]

Another use case for this is to study the effects of DVR and to create some DVR benchmarks (for CTAO, we'd add them to the datapipe-testbench, but it would need a file with at least some R1 and DL0 as inputs)