Unable to run Text to Image and Image to Text models on NPU. [NPU Excluded]

[Harsha0056]

I reviewed the notebooks for both image generation and multimodal tasks, and in all of them, I noticed that the NPU is either not detected or excluded from selection when detected. I want to run these models on the NPU.

Is there an alternative approach I can try, or should I wait for the next update? Could there be something wrong with my current method? Please assist me with this. Thank you.

[brmarkus]

Can you provide more details, please?
Do you use a specific JupyterNotebook from this model?
Which models do you mean?
What does your environment look like (which OS, which version of the OS, which version of Pythin, natively or using a Python virtual environment or Conda)?
Which HW are you running on? Which version of this repo are you using, have you changed source code, have you updated your dependencies in case you installed the "requirements.txt" a while ago?

Can you confirm whether the used models make use of dynamic shapes?

[Harsha0056]

https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/phi-3-vision/phi-3-vision.ipynb

in line number 6 we can see that npu is being excluded

[brmarkus]

Initially many Jupyter notebooks have NPU excluded when dynamic (input- and/or output-shapes) are involved. For some models it's possible to manually with changes (source code and/or model) get it working for NPU, but not for all.

Let's see what the OpenVINO-Notebooks development-team says.

[Harsha0056]

I’ve observed this issue across several notebooks, except for an RAG-based notebook where the NPU was included for the embedding model.

https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-rag-langchain/llm-rag-langchain-genai.ipynb

Line no 18

[brmarkus]

Yes, typically this is the case for models using dynamic shapes; but many (the majority?) models are supported and the NPU can perfectly used for e.g. offloading CPU and GPU.
See the NPU plugin documentation, section "Limitations":

"https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.html#limitations"

Limitations
Currently, only models with static shapes are supported on NPU.

Import/Export:

Offline compilation and blob import is supported only for development purposes. Pre-compiled models (blobs) are not recommended to be used in production. Blob compatibility across different OpenVINO / NPU Driver versions is not guaranteed.

[Harsha0056]

https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-rag-langchain/llm-rag-langchain-genai.ipynb

Line no 18

In this notebook they mentioned it for embedding model

[brmarkus]

When pipelines consist of multiple models, then one or some or the models could "run on the NPU", wile the others run on other devices (CPU, GPU, VPU, FPGA, AUTO, MULTI, HETERO).

Let's see of someone from the developer team can comment, maybe it is outdated or a left-over?