Performance CBF vs. HDF5

With thanks to Martin Savko from Soleil for provision of data in both HDF5 and CBF format:

• 20 cores/node
• 20 nodes

i.e. standard set up 1800 frame set processed in ~ minute in both formats:

Fast_DP installed in: /dls_sw/apps/fast_dp/2017_08_10
Starting image: /dls/mx-scratch/gw56/Eiger/Soleil/transthyretin/h5/200Hz_0.1dpf_1_master.h5
Reading /dls/mx-scratch/gw56/Eiger/Soleil/transthyretin/h5/200Hz_0.1dpf_1_master.h5 took 8.45s
Running on: cs04r-sc-com13-16
Number of jobs: 20
Number of cores: 20
Processing images: 1 -> 1800
Phi range: 0.00 -> 180.00
Template: 200Hz_0.1dpf_1_??????.h5
Wavelength: 0.98012
Working in: /dls/tmp/gda2-fdp
All autoindexing results:
Lattice      a      b      c  alpha   beta  gamma
     oP  43.20  64.30  85.30  90.00  90.00  90.00
     mP  43.20  64.30  85.30  90.00  90.00  90.00
     aP  43.20  64.30  85.30  90.00  90.00  90.00
Mosaic spread: 0.14 < 0.20 < 0.29
Happy with sg# 16
 43.20  64.30  85.30  90.00  90.00  90.00
--------------------------------------------------------------------------------
      Low resolution  28.42  28.42   1.74
     High resolution   1.70   7.59   1.70
              Rmerge  0.035  0.019  0.898
             I/sigma  24.30  71.50   1.50
        Completeness   99.7   96.4   97.1
        Multiplicity    6.6    5.6    5.9
              CC 1/2  0.999  0.999  0.691
  Anom. Completeness   99.3   98.5   94.0
  Anom. Multiplicity    3.4    3.5    2.9
   Anom. Correlation -0.056 -0.099 -0.052
               Nrefl 178382   1946  11342
             Nunique  26923    346   1920
           Mid-slope  0.933
                dF/F  0.040
          dI/sig(dI)  0.765
--------------------------------------------------------------------------------
Merging point group: P 2 2 2
Unit cell:  43.21  64.36  85.26  90.00  90.00  90.00
Processing took 00h 00m 53s (53 s) [178382 reflections]
RPS: 3360.7

vs.

Fast_DP installed in: /home/gw56/svn/fast_dp
Starting image: ../cbf/200Hz_0.1dpf_1_00001.cbf
Running on: cs04r-sc-com99-07
Number of jobs: 20
Number of cores: 20
Processing images: 1 -> 1800
Phi range: 0.00 -> 180.00
Template: 200Hz_0.1dpf_1_#####.cbf
Wavelength: 0.98010
Working in: /dls/mx-scratch/gw56/Eiger/Soleil/transthyretin/sci-6024
All autoindexing results:
Lattice      a      b      c  alpha   beta  gamma
     oP  43.20  64.30  85.30  90.00  90.00  90.00
     mP  43.20  64.30  85.30  90.00  90.00  90.00
     aP  43.20  64.30  85.30  90.00  90.00  90.00
Mosaic spread: 0.14 < 0.20 < 0.29
Happy with sg# 16
 43.20  64.30  85.30  90.00  90.00  90.00
--------------------------------------------------------------------------------
      Low resolution  28.42  28.42   1.74
     High resolution   1.70   7.59   1.70
              Rmerge  0.035  0.018  0.899
             I/sigma  24.40  71.70   1.50
        Completeness   99.7   96.4   97.0
        Multiplicity    6.6    5.6    5.9
              CC 1/2  1.000  0.999  0.686
  Anom. Completeness   99.3   98.5   94.0
  Anom. Multiplicity    3.4    3.5    3.0
   Anom. Correlation -0.081 -0.218 -0.040
               Nrefl 178369   1944  11340
             Nunique  26919    346   1918
           Mid-slope  0.932
                dF/F  0.040
          dI/sig(dI)  0.766
--------------------------------------------------------------------------------
Merging point group: P 2 2 2
Unit cell:  43.21  64.36  85.26  90.00  90.00  90.00
Processing took 00h 01m 05s (65 s) [178369 reflections]
RPS: 2724.2

This is using the "dectris-neggia.so" plugin. Absence of substantial performance penalty combined with the file system saving (4GB for HDF5 vs. 18 GB for CBF; both compressed with bz2 around 3.0GB) closes out any remaining disadvantages to using this format...

Thanks to Kay Diederichs for help with the forkxds scripts; Wolfgang Kabsch for providing the mechanism to make this possible once more... you will need BUILT=20170720 or later for this to work correctly.