multiple preprocessing using bioimageio spec

[pr4deepr]

Hi
I am duplicating an imagesc forum post as I think its better to ask the question here.
I am trying to use preprocessing in the new version of deepimageJ. Can I chain multiple preprocessing steps in the bioimageio model spec?

I have a pytorch model that worked on deepimagej v3.0.3. I used the preprocessing macro for fixed_zero_mean_unit_variance, but modified it to use

paramMean = newArray(0.485, 0.456, 0.406);
paramStd = newArray(0.229, 0.224, 0.225);

This was working well until the new update where I cannot use a custom macro anymore.
I can use the zero_mean_unit_variance preprocessing in the rdf.yaml file like:

inputs:
- axes: bcyx
  data_range: [0.0, 255.0]
  data_type: uint8
  name: input
  preprocessing:
  - name: zero_mean_unit_variance
    kwargs:
      axes: yx
      mean: [0.485, 0.456, 0.406]
      mode: fixed
      std: [0.229, 0.224, 0.225]

but the model prediction is wrong.

Using the deepimagej v3.1.0, if I use the above configuration, I get the prediction in the middle (wrong). However, if I remove preprocessing config, run the fixed_zero_mean_unit_variance.ijm macro first and then predict with deepimagej, I get the image on the right (correct).

So, my question is, without needing to use a macro, how can I leverage the model spec to:

• convert image to RGB stack
• convert to 32-bit
• follow up with zero mean unit variance normalization as above.

Cheers
Pradeep

[FynnBe]

I can only give answers to the general spec related aspects of your question... This repo is not deepimagej specific, but maybe @carlosuc3m can chime in on this here, too. Otherwise please open another issue in https://github.com/deepimagej/deepimagej-plugin/issues

convert image to RGB stack

The spec does not distinguish between an image and an RGB stack, for it all n-dimensional data are tensors. An input tensor can have a channel dimension with channel names determining the number of channels. (Python developer docs, non Python (JSON Schema docs))

convert to 32-bit

Conversion to float32 before and after preprocessing is the default when specifying dtype: float32 for an input tensor. (unless the last perprocessing step is binarize). For more complex situations (that you don't seem to have, I only mention it for completeness, there is a ensure_dtype preprocessing step.

Follow up with zero mean unit variance normalization as above.

The spec looks good in the example you gave 👍
I do not know how deepimagej ingests the spec though... @carlosuc3m can probably help here.

[pr4deepr]

Thanks for that.

I forgot to add, there is an extra normalization step, i.e., divide by 255, so the procedure is:

• convert image to RGB stack
• convert to 32-bit
• normalization: divide by 255
• follow up with zero mean unit variance normalization as above.

It looks like other steps are fine, so is the normalization part of preprocessing or do I need to add it as an extra step before 'zero mean unit variance'?

Cheers
Pradeep

[carlosuc3m]

Hello @pr4deepr ,
Yes you can stack pre-processing steps in dij, as long as they are th eones supported by th ebioimage.io.
CAn you please describe hte whole pre-processing pipeline?
I will try to implement it and see if the result works.

Also as @FynnBe says, no need to worry about composite images, rgb stacks or data type! All that is handled by deepimagej

[pr4deepr]

Thanks @carlosuc3m. I am attaching the preprocessing macro that worked for me in deepimagej 3.0.3.

fixed_zero_mean_unit_variance.txt

• convert to 32 bit
• divide each channel (if multichannel) by 255
• apply zero mean unit variance norm using mean of (0.485, 0.456, 0.406) and std of (0.229, 0.224, 0.225)

Cheers
Pradeep

[pr4deepr]

@carlosuc3m , just following up on this

[carlosuc3m]

Hello @pr4deepr and sorry for the delay in the response!!
Specifying the pre-processing like this should make deepiumagej to run one transfor mation after the other.

HEre note that in scale_linear, gain = 1/255.

By the way, in the macro you provided, you first divide teh channels by 255, but multiply the mean and std by 255...
Maybe I have not understood well!!

REgards,
CArlos

inputs:
- axes: bcyx
  data_range:
  - -.inf
  - .inf
  data_type: float32
  name: raw
  preprocessing:
  - kwargs:
      axes: xy
      eps: 1e-06
      gain: 0.00392156862
      mode: per_sample
    name: scale_linear
  - kwargs:
      axes: xy
      eps: 1e-06
      mean:
      - 0.485
      - 0.456
      - 0.406
      std:
      - 0.229
      - 0.224
      - 0.225
      mode: fixed
    name: zero_mean_unit_variance

[pr4deepr]

Thanks.

I get an error with scale_linear

java.lang.IllegalArgumentException: Setter 'setMode' should have only one input parameter with type Object.class.
	at io.bioimage.modelrunner.model.processing.TransformationInstance.checkArgType(TransformationInstance.java:281)
	at io.bioimage.modelrunner.model.processing.TransformationInstance.setArg(TransformationInstance.java:238)
	at io.bioimage.modelrunner.model.processing.TransformationInstance.createInstanceWithArgs(TransformationInstance.java:222)
	at io.bioimage.modelrunner.model.processing.TransformationInstance.build(TransformationInstance.java:167)
	at io.bioimage.modelrunner.model.processing.TransformationInstance.<init>(TransformationInstance.java:63)
	at io.bioimage.modelrunner.model.processing.TransformationInstance.create(TransformationInstance.java:78)
	at io.bioimage.modelrunner.model.processing.Processing.buildPreprocessing(Processing.java:79)
	at io.bioimage.modelrunner.model.processing.Processing.<init>(Processing.java:66)
	at io.bioimage.modelrunner.model.processing.Processing.init(Processing.java:110)
	at io.bioimage.modelrunner.model.BioimageIoModel.runBMZ(BioimageIoModel.java:357)
	at io.bioimage.modelrunner.model.BioimageIoModel.run(BioimageIoModel.java:332)
	at deepimagej.Runner.runOnTestImages(Runner.java:146)
	at deepimagej.gui.Gui.lambda$runModelOnTestImage$22(Gui.java:314)
	at java.lang.Thread.run(Thread.java:750

When looking at the spec, I couldn't fine an entry for eps for scale_linear?

[carlosuc3m]

Hello @pr4deepr
Could you please update deepimagej witht he update site?

It should work now. You can also remove the parameters mode and eps, but it should not matter.

For compatibility with other softwares also add offset: 0

inputs:
- axes: bcyx
  data_range:
  - -.inf
  - .inf
  data_type: float32
  name: raw
  preprocessing:
  - kwargs:
      axes: xy
      gain: 0.00392156862
      offset: 0.0
    name: scale_linear
  - kwargs:
      axes: xy
      eps: 1e-06
      mean:
      - 0.485
      - 0.456
      - 0.406
      std:
      - 0.229
      - 0.224
      - 0.225
      mode: fixed
    name: zero_mean_unit_variance

REgards,
Carlos

[pr4deepr]

Thanks. So, it runs fine, but the prediction is not accurate in my case.
If I convert to 32 bit and divide by 255, before applying the deepimagej model using the configuration you've given me, then the prediction is accurate.

Maybe this could be something to do with how I've performed normalization in my code during training

I checked my code and during training in pytorch, following preprocessing happens:

• load images as float32
• divide by 255
• Augmentation using albumentations normalize
  In this function above, max pixel value argument is 255.0 by default. I haven't changed this. This may explain why division by 255 first, followed by multiplying the mean and std by 255 worked in the macro I shared.

Perhaps I could modify the mean and std.. Or maybe there is another fix?
How can I verify that scale_linear is working as intended?

Cheers
Pradeep