Add vit attribution hugo

[HugoDeBosschere]

Description

Added a small MVP for ViT attribution. For now no vizualisation but there is a Saliency method that executes.
I did not modify any preexisting code, it's only bonus.

Creation of:

1. ImageGranularity mirroring Granularity with PATCH (default) and PIXEL granularities in interpreto/commons/image_granularity.py
2. ImageClassificationInferenceWrapper in interpreto/model_wrapping/image_classification_inference_wrapper.py.
   It returns a flattened 2D tensors to be able to reuse for free the methods developped for text. In this class 3 methods were overriden:
   • init
   • _prepare_inputs
   • _compute_gradients
3. ImageAttributionOutput and ImageAttributionExplainer in interpreto/attributions/image_base.py. ImageAttributionOutput has the same field as AttributionOutput except granularity is ImageGranularity and it stores the raw_image in raw_image for vizualisation purposes. ImageAttributionExplainer does almost exactly the same things as AttributionExplainer but some name change to adapt to image language and the explain function stores the raw_image in the raw_image field of AttributionOutput
   4.ImagePerturbator in interpreto/attributions/perturbations/image_base.py. Only the base class for the MVP so the perturbator is a no-op. Since we don't need to check for model_inputs or model_embeds it just returns the intputs. Allows to instantiate ImageSaliency to check if the pipeline runs.
4. ImageSaliency in interpreto/attributions/methods/image_saliency.py. Same as Saliency but tokenizer -> image_processor
5. first_tests/first_test_image.py to test the whole pipeline with a cat image. The cat image is not provided (sorry)

• 🚀 New feature (non-breaking change which adds functionality)

Checklist

• [ x] I've read the CODE_OF_CONDUCT.md document.
• [x ] I've read the CONTRIBUTING.md guide.
• I've successfully run the style checks using make lint.
• I've written tests for all new methods and classes that I created and successfully ran make test.
• I've written the docstring in Google format for all the methods and classes that I used.

[AntoninPoche]

The commit messages are clear, so I can review them commit by commit. Its nice! I guess thanks, Claude ^^

Overall, it is clear, and it is nice that you already managed to have something running. My main comment is that ImageGranularity is far too complex for what you need for images. But you will really know after trying perturbations at both granularities.

In any case, you can discuss all comments; they are my opinions, not based on having tried the thing myself.

[AntoninPoche]

Are you sure preprocessing by default is the way to go?

[HugoDeBosschere]

I would have said yes because in my head the typical use case would be using this on a PIL image where this requires preprocessing. Of course maybe I'm wrong and it will be more generally used in already preprocessed tensors in which case it may be better to put preprocess = False.
I think this has to be resolved by how it will most often be used, which I have little visibility on.

[AntoninPoche]

What happens if we process already processed inputs? Does it do something weird?

If so, then it might be tricky, otherwise, default preprocess is the way to go.

[HugoDeBosschere]

In most cases it will do something weird indeed. But if the user does not preprocess the inputs it also does something weird (the ViT will not function as expected). So I don't know which one is best.

[AntoninPoche]

The current behavior looks good to me. It is just that I was not familiar with how things work in ViTs, and I wanted the default choice to preprocess or not to be made consciously.

[AntoninPoche]

You cannot inherit directly from the ClassificationAttributionExplainer class. In particular, if your goal is just to reuse the target's preprocessing.

The goal is to find a way to use or adapt the MultitaskExplainerMixin that kind of serves as a factory. When you instantiate Saliency, it checks the model class and assigns the base task-explainer to inherit from both and have what you need.

The thing is that this hinders clarity. IMO, they should all be at the same level.

[AntoninPoche]

But for now, it can be kept like this if you want easy tests.

[AntoninPoche]

Not sure what you use this for.

This was used in the text to know to which words or tokens attributions correspond.

For images, you can just store the images in their original (h, w) and the attributions in the granularity (h, w). Then, for visualizations, we resize (which does nothing for pixels and something for patches). (Note that the way of resizing can be a visualization parameter.)

In summary, if you already store the image, I would set None for the elements or omit it.

[AntoninPoche]

We discussed and this outdated

[AntoninPoche]

You do not implement self.post_processing. However, the parent class just does return ModelTask.CLASSIFICATION, contribution.

I do not think you should use the same ModelTask, at least for now.

So you can remove this line and just create an ImageClassification task to pass to your ImageAttributionOutput. Or just omit it.

If you omit everything you do not need, it might be easier to identify what's common and not for later merging.

[HugoDeBosschere]

Okay I removed it.

[AntoninPoche]

There should be a simpler solution than 4 nested for loops.

In any case, my first intuition is that you do not the get_indices. At least for the two granularities you have here.

The granularity is used at two steps:

• For perturbations, either your perturb the pixel_values or the patch so no need for granularity.
• For Aggregation if your attributions are not already at the correct granularity. In this case, for now, you just have the case were you request patch for gradients where you need to downscale the attributions from pixel-wise to patch-wise.

So IMO, you do not need most of the granularity elements for images. It is useful for text because each sample has a different size, with words and sentences distributed across the whole sample. It is much more consistent with images.

Well, the above comments do not make sense if you have special patch/tokens which do not correspond to pixels but to constants (to my understanding, these are used in ViTs and such).

[AntoninPoche]

Partially outdated

[AntoninPoche]

Why flatten? I know it breaks the shape from text, but it makes much more sense to have an (H,W) shape IMO.

This depends on how the rest is done.

[AntoninPoche]

You told it was for aggregation right?

[HugoDeBosschere]

This allows us to reuse aggregator entirely without changing anything. It also allowed me to copy with minimal changes a lot of the functions that were already written for text (such as the perturbators for example, or the granularity) so that we may more easily refactor after.

[AntoninPoche]

Then it is a good choice!

[AntoninPoche]

I imagine that for images, this should correspond to the interpolation mode.

[HugoDeBosschere]

I'm not sure if I correctly understand the comment but granularity_aggregation_strategy is the strategy used when use_gradient = True to go from pixel scores to granularity scores.

If the interpolation mode you reference here is the one in the aggregator then, from what I understand, this aggregates on the perturbations, not the granularity.

So I'm not sure about what to respond.

[AntoninPoche]

I am talking about the granularity of the aggregation here.

After gradients on the pixel_values, if you want to go to the patch granularity, you need to resize the explanations.

The image resizing operation takes an interpolation parameter. This parameter is a kind of choice we make with the GranularityAggregationStrategy. Well, you could also call this pooling, but resize allows you to think at the image level.

[HugoDeBosschere]

Okay I undestand better.
Interpolation and pooling both work but I have a preference for pooling since this looks a lot like your typical max pooling operation (when the strategy is MAX) and those terms are generally used in CNNs (here it would be a patch_size * patch_size pooling operation). If you think interpolation is better I can put interpolation.

[AntoninPoche]

I think resizing is better adapted, indeed, because it considers the image as a whole. While pooling makes small windows and extracts a value from them.

Here, taking the maximum, mean, or any other value does not really make sense. Well, it kind of does the job, but resizing is a much better way to change an image's size. (Which also applies to attributions with shape (h, w)).

Some methods, like the CAM family or Rise, provide smooth, human-friendly attributions as a result.

[HugoDeBosschere]

Ok I'll look into replacing GranularityAggregationStrategy with an InterpolationStrategy and to define several stategies of interpolation

[AntoninPoche]

We should discuss with @fanny-jourdan if we want torchvision to be a main dependency or in something like pip install interpreto[vision].

[AntoninPoche]

I think it should be harmonized before being saved. It will be much easier for visualizations.

So it should have a single type.

[HugoDeBosschere]

I had kept it this way because I didn't want to plot a modified image for the user (I thought it was best if he saw the same image he had first loaded). I already did the visualization script so : do I still need to modify this field ?
If the concern was that the visualization was going to be hard then maybe I can keep it.
If it was simplicity then I can just use model_inputs_to_explain as the base image onto which to display the heatmap.

[AntoninPoche]

What you are saying is that once processed, if we plot the image, it looks weird, so we need to keep a raw image, right?

If that is the case, I agree with you.

[HugoDeBosschere]

I haven't actually tried to plot it with the processed image. Since the comment had not been submitted before I actually wrote the visualization functions, I used the raw_image and it works. Do you want me to try and see if I can also make it work with the processed image so that we may drop this field that may not be useful ?

[AntoninPoche]

I think this should go in process_model_inputs. So you know that you return None if inputs are already processed.

[AntoninPoche]

Hi @HugoDeBosschere, with @thomas-mullor we discussed how to tackle the perturbators problem.

@fanny-jourdan do not hesitate to give your opinion.

To summarize, the problem was that perturbators depend both on the method and the modality.
But we do not want to implement each combination of method and modality.

So we found a way to dynamically create the necessary perturbator by inheriting from both the method and modality specific classes.

Here is a summary of the modifications:

# attributions/perturbations/base.py

class Perturbator(ABC):
    @abstractmethod
    def perturb(self, inputs):
        pass

class TensorPerturbator(Perturbator):  # new class (just for typing and clarity)
    @abstractmethod
    def perturb_tensor(self):  # renaming of `perturb_embeds`
        pass

class TextTensorPerturbator(TensorPerturbator):  # renaming of `EmbeddingsPerturbator`
    def perturb(self, inputs):  # already exists
        # calls `perturb_tensor`

class ImageTensorPerturbator(TensorPerturbator):  # renaming of your `ImageEmbeddingsPerturbator`
    def perturb(self, inputs):  # already exists but can surely be simplified
        # calls `perturb_tensor`

class MaskPerturbator(Perturbator):  # new class (just for typing and clarity)
    @abstractmethod
    def get_mask(self):
        pass

class TextMaskPerturbator(MaskPerturbator):  # renaming of `IdsPerturbator`
    def perturb(self, inputs):  # already exists
        # calls `get_mask`

class ImageMaskPerturbator(MaskPerturbator):  # renaming of your `ImageIdsPerturbator`
    def perturb(self, inputs):  # already exists
        # calls `get_mask`

# attributions/perturbations/occlusion_perturbation.py

from .base import MaskPerturbator

class OcclusionPerturbator(MaskPerturbator):  # change inheritance, might make the type checker unhappy
    def get_mask(self, mask_dim):
        # should support both `TextMaskPerturbator` and `ImageMaskPerturbator` requirements
        # it might be nothing at first

# attributions/perturbations/gaussian_noise_perturbation.py

from .base import TensorPerturbator

class GaussianNoisePerturbator(TensorPerturbator):  # change inheritance, might make the type checker unhappy
    def perturb_tensor(self, inputs_embeds):
        # should support both `TextTensorPerturbator` and `ImageTensorPerturbator` requirements
        # therefore, support (1, l, d) and (1, c, h, w) shapes
        # (1, ...) -> (p, ...)

# attributions/base.py

from .perturbations.base import (
    Perturbator,
    TensorPerturbator,
    TextTensorPerturbator,
    ImageTensorPerturbator,
    MaskPerturbator,
    TextMaskPerturbator,
    ImageMaskPerturbator
)

class AttributionExplainer(ABC):
    associated_inference_wrapper: InferenceWrapper
    base_tensor_perturbator_class: type[TensorPerturbator]
    base_mask_perturbator_class: type[MaskPerturbator]

    # does not impact the methods

class TextClassificationExplainer(AttributionExplainer):
    associated_inference_wrapper: TextClassificationInferenceWrapper
    base_tensor_perturbator_class = TextTensorPerturbator
    base_mask_perturbator_class = TextMaskPerturbator

class TextGenerationExplainer(AttributionExplainer):
    associated_inference_wrapper: TextGenerationInferenceWrapper
    base_tensor_perturbator_class = TextTensorPerturbator
    base_mask_perturbator_class = TextMaskPerturbator

class ImageClassificationExplainer(AttributionExplainer):
    associated_inference_wrapper: ImageClassificationInferenceWrapper
    base_tensor_perturbator_class = ImageTensorPerturbator
    base_mask_perturbator_class = ImageMaskPerturbator

class MultitaskExplainerMixin:
    # no modifications specific to the new perturbator classes
    # still, it should include the ImageClassificationExplainer at some point

# attributions/methods/occlusion.py

class Occlusion(MultitaskExplainerMixin, AttributionExplainer):
    def __init__(...):
        # create the perturbator dynamically by inheriting from both the method and modality specific classes
        perturbator_class = type(
            "ModalitySpecific" + self.__class__.__name__,  # name
            (OcclusionPerturbator, self.base_mask_perturbator_class,),  # parent classes
            {}
        )
        perturbator = perturbator_class(
            tokenizer=tokenizer,
            granularity=granularity,
            replace_token_id=replace_token_id,
        )
        ...

# attributions/methods/smooth_grad.py

class SmoothGrad(MultitaskExplainerMixin, AttributionExplainer):
    def __init__(...):
        # create the perturbator dynamically by inheriting from both the method and modality specific classes
        perturbator_class = type(
            "ModalitySpecific" + self.__class__.__name__,  # name
            (GaussianNoisePerturbator, self.base_tensor_perturbator_class,),  # parent classes
            {}
        )
        perturbator = perturbator_class(
            inputs_embedder=model.get_input_embeddings(),
            n_perturbations=n_perturbations,
            std=noise_std
        )
        ...