Processing-friendly HDF5 master file structure

[marcus-oscarsson]

Updating this with some comments from @pkeller (thanks :) ), who suggested:

Discussions about HDF5 master file structure and metadata for complex data collections. Examples include:

• characterisation data collected in multi-trigger mode

• wavelength-interleaved and inverse beam data collections

[pkeller]

An example of the problems with multi-trigger mode is seen with a dataset collected in June 2024 at id30b:

Just to let you know that Marcus and Jeremy managed to get a 'fast
characterisation' scan implementation to run (4 wedges of 10 images, 0.1
deg oscillation at 0, 90, 180 and 270). All the images were put into a
single file that you can find in the directory below:

https://oc.embl.de/index.php/s/YLMxXtl5taPMziM

The omega angles in the metadata are as follows:

% h5ls -d Sample-8-2-031_master.h5/entry/sample/goniometer
omega                    Dataset {40}
    Data:
         0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39
omega_end                Dataset {40}
    Data:
         1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40
omega_range_average      Dataset {SCALAR}
    Data:
         1
omega_range_total        Dataset {SCALAR}
    Data:
         40

There are two things wrong with this:

• The frame widths should be 0.1°, not 1°
• The omega angles should not be continuous, but reflect the fact that there are large gaps.

IIUC the metadata should be as follows:

omega                    Dataset {40}
    Data:
         0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 90.0 90.1 90.2 90.3 90.4 90.5 90.6 90.7 90.8 90.9 180.0 180.1 180.2 180.3 180.4 180.5 180.6 180.7 180.8 180.9 270.0 270.1 270.2 270.3 270.4 270.5 270.6 270.7 270.8 270.9
omega_end                Dataset {40}
    Data:
         0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 90.1 90.2 90.3 90.4 90.5 90.6 90.7 90.8 90.9 91.0 180.1 180.2 180.3 180.4 180.5 180.6 180.7 180.8 180.9 181.0 270.1 270.2 270.3 270.4 270.5 270.6 270.7 270.8 270.9 271.0
omega_range_average      Dataset {SCALAR}
    Data:
         0.1
omega_range_total        Dataset {SCALAR}
    Data:
         271

At Soleil, the omega angles are fixed up after collection using an h5py script: https://github.com/MartinSavko/experimental_methods/blob/8b9e9a9d38baad413ee5306f2ed9a74987f9277e/reference_images.py#L273

Other workarounds are of course possible, but in an ideal world the metadata would be correct in the HDF5 metadata as first written.

@marcus-oscarsson has confirmed verbally that at the time of writing it is not known what the omega angles would be if this data collection were to be repeated at id30b now.

Max Burian at DECTRIS is aware in general terms that there are issues in this area with HDF5, and we at GPhL plan to raise them with him and Sofia Trampari in the follow-up initiative to HDRMX started by Graeme Winter.