JPEG-2000 Lossy quality seems to have a low ceiling

[xinaesthete]

Rather than as described

We recommend experimenting with different quality values between 0.25 and the bit depth of the input image to find an acceptable tradeoff between file size and visual appeal of the converted images.

I find that increasing quality beyond a relatively low number doesn't result in a larger file, and still a noticeable amount of artefacts in the output. For example, I've been testing on an image that is ~7.3gb with lossless, but seems to end up around 125mb. Actually, it still looks remarkably good and I'll probably work with a few things at that level, but would probably want to tune it somewhat higher.

I am running v 0.7.1 on MacOS, installed with Mamba. Actually, I haven't had time to test extensively (and am about to go on leave for a few weeks), but I possibly saw different behaviour with 0.7.0 on a cluster - seem to recall the size did keep increasing as expected... I have another issue with checking the output as the images were float32 though, which given I'm decoding with openjpeg for visualisation means I'd need some quantization step. I digress.

[melissalinkert]

If you have reason to suspect that the quality option isn't working as intended, then it would definitely help to have as many concrete details as possible, since compression performance (both in terms of file size and visual appearance) are dependent upon the input data. If any other lossy compression was performed on the data prior to raw2ometiff, that can also impact the appearance of artifacts in the image.

Can you let us know how the input dataset was generated (i.e. the specific tool and options used)? Are you able to share this data publicly? What is the specific raw2ometiff command that was used (including all options)?

[xinaesthete]

I'm away for a few weeks, but will do when I'm back in September.

[xinaesthete]

What I can say is that one example I processed was morphology_focus from a public xenium dataset from 10x, run first through bioformats2raw.

https://www.10xgenomics.com/datasets/xenium-prime-ffpe-human-ovarian-cancer

[xinaesthete]

Options for raw2onetiff were just compression "JPEG 2000 Lossy" and various values of quality like 0.25, 16, 1e11... They didn't all come out identical, but as I say, appeared to level out at quite a low ceiling.

[melissalinkert]

With --compression "JPEG-2000 Lossy", the --quality option is expected to have the most effect if set below the number of bits required to store one pixel (e.g. 16 for uint16 input data). Any values of --quality at or above the number of bits per pixel are expected to produce nearly indistinguishable output. This is mentioned in the README here:

https://github.com/glencoesoftware/raw2ometiff/blob/master/README.md?plain=1#L102-L104

If the original input data was lossily compressed, I would expect that there will be some artifacts as a result, independent of the compression options selected in raw2ometiff. It would be useful to know if the same pattern of artifacts appears when fully lossless compression is used (e.g. --compression LZW or --compression JPEG-2000 with no --quality).