Support for GPU in llava's CLIP image encoder

[y10ab1]

To the best of my knowledge, the image encoder of CLIP does not currently support GPU. This presents a bottleneck for VQA and image captioning.

Is it possible to enable GPU support for the CLIP image encoder?
I think this could enhance the response speed for multi-modal inferencing with llama.cpp.

[cmp-nct]

It appears all required GPU operations are now supported, if you look at "Issues" you'll see it's already a topic
CLIP is currently quite a considerable factor when using llava, takes about 500-700ms to calculate CLIP embeddings compared to a few ms when using python transformer.

[Fatemehkiasaveh]

how can I install it on gpu ? I have image-text to text model pytorch model I want to use CLIP to convert it to onnx is this possible?

[monatis]

Track the progress in #4061