As we explore tools to better compress GEOS NetCDF4 output, one interesting set of tools looks to be the pressio suite of @robertu94:
https://github.com/robertu94/libpressio
https://github.com/robertu94/pressio-tools
where the latter has a CLI:
pressio -i ~/git/datasets/hurricane/100x500x500/CLOUDf48.bin.f32 \
-b compressor=sz3 -o abs=1e-4 -O all \
-m time -m size -m error_stat -M all \
-w /path/to/output.dec
I'm really interested in that if we use Spack, we can define a lot of variants to let us test:
Variants:
arc [false] false, true
build support for the ARC error correction tool
bitgrooming [false] false, true
build support for the bitgrooming
blosc [false] false, true
support the blosc lossless compressors
boost [false] false, true
support older compilers using boost
build_system [cmake] cmake
Build systems supported by the package
bzip2 [false] false, true
build support for the bzip2 compressor
core [true] false, true
build core builtin libraries
cuda [false] false, true
Build with CUDA
digitrounding [false] false, true
build support for the digit rounding
docs [false] false, true
build and install manual pages
fpzip [false] false, true
support for the FPZIP lossy compressor
ftk [false] false, true
build support for the feature tracking toolkit
hdf5 [false] false, true
support reading and writing from hdf5 files
json [false] false, true
build the JSON support
libdistributed [false] false, true
support for distributed multi-buffer support
lua [false] false, true
support for composite metrics using lua
magick [false] false, true
support the imagemagick image compressors
mgard [false] false, true
support for the MAGARD error bounded lossy compressor
mgardx [false] false, true
build support for the MGARDx compressor
mpi [false] false, true
support for launching processes using mpi
ndzip [false] false, true
build support for the NDZIP compressor
netcdf [false] false, true
build support for the NDFCDF data format
openmp [false] false, true
build plugins that use openmp
petsc [false] false, true
support IO using petsc format
python [false] false, true
build the python wrappers
qoz [false] false, true
build support for the qoz compressor
remote [false] false, true
build the remote launch plugin
sz [false] false, true
support for the SZ error bounded lossy compressor
sz3 [false] false, true
build support for the SZ3 compressor family
szauto [false] false, true
build szauto support
unix [false] false, true
build support for unixisms like mmap and rusage
zfp [false] false, true
support for the ZFP error bounded lossy compressor
Thanks to @robertu94 for adding so many. Now the spack packages do seem a bit out of date (there is a pressio 1.0.4 and 0.5.3 currently but those aren't in spack), but I can add them when I test (and then submit up to spack-packages).
As we explore tools to better compress GEOS NetCDF4 output, one interesting set of tools looks to be the pressio suite of @robertu94:
https://github.com/robertu94/libpressio
https://github.com/robertu94/pressio-tools
where the latter has a CLI:
I'm really interested in that if we use Spack, we can define a lot of variants to let us test:
Thanks to @robertu94 for adding so many. Now the spack packages do seem a bit out of date (there is a pressio 1.0.4 and 0.5.3 currently but those aren't in spack), but I can add them when I test (and then submit up to spack-packages).