[ITensors] [ENHANCEMENT] Support HDF5 compression

[liuzhaochen]

Dear ITensor developers,

I noticed that HDF5.jl provides various methods for compressing datasets.

I wonder whether it would be possible to support saving ITensors to HDF5 with compression directly.

Thank you for your attention.

Best regards,
Zhaochen

[mtfishman]

I don't know much about compression in HDF5, I guess you mean: https://juliaio.github.io/HDF5.jl/stable/#Chunking-and-compression? It looks like those are controlled with extra keyword arguments to write, I think we could allow that just be forwarding arbitrary keyword arguments down to where we are writing the tensor data: https://github.com/ITensor/ITensors.jl/blob/v0.9.15/NDTensors/ext/NDTensorsHDF5Ext/dense.jl#L11. However, it seems like it requires manually specifying some chunk sizes so that doesn't look so convenient to use.

EDIT: It looks like there is a known bug where if you use write and specify compress, it also requires that you specify chunk (JuliaIO/HDF5.jl#822).

[mtfishman]

I can make a quick PR to forward keyword arguments through our write definitions, would that be sufficient for now?

[liuzhaochen]

Hi Matt,

Thanks for the updates. I’ve actually tested the compression provided by HDF5.jl, but I found that it doesn’t reduce the size of the final file. It’s likely that lossless compression algorithms are not very effective for irregular Float numbers.

Anyway, thanks for your input.

Best,
zc liu