Skip to content

[NPU] Support NPUW for text-embedding models #3244

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mengweiguo wants to merge 18 commits into
base: releases/2026/0
Choose a base branch
from
+782 −222
Open

[NPU] Support NPUW for text-embedding models #3244

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
e9d9704
Support NPUW for text-embedding models
mengweiguo Dec 4, 2025
864c23d
Fix review remark
mengweiguo Dec 4, 2025
fdaaeed
Add option `normalize` support
mengweiguo Dec 5, 2025
1d0db49
Add tests
mengweiguo Dec 7, 2025
b417b56
tests fallback to CPU
mengweiguo Dec 10, 2025
0593b5f
Handle the post-processing with a separate model
mengweiguo Dec 12, 2025
0a10db2
Add qeury tests
mengweiguo Dec 21, 2025
d193a10
Fix corner case and add comments
mengweiguo Dec 23, 2025
52510bf
Change `embedding_pad_to_max_length` to `disable_embedding_pad_to_max…
mengweiguo Jan 8, 2026
25dcae1
Fix AI review
mengweiguo Jan 9, 2026
b5f6660
Fix code style
mengweiguo Jan 14, 2026
0789e60
Fix Lint and tests
mengweiguo Jan 14, 2026
41ac8c9
Refactor code
mengweiguo Jan 15, 2026
d7976c0
Restore `emb_pad_to_max_length`
mengweiguo Jan 16, 2026
c100e1c
Set `padding` according to `pad_to_max_length`
mengweiguo Jan 19, 2026
c49ffae
Resolve review remarks
mengweiguo Jan 21, 2026
affc53e
Move `npu/text_embedding_pipeline.hpp`
mengweiguo Jan 21, 2026
2eb3cf1
Move property description to public header
mengweiguo Jan 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions src/cpp/include/openvino/genai/rag/text_embedding_pipeline.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,20 @@
namespace ov {
namespace genai {

//
// Enable NPU dynamic prompt input support, which allows text-embedding models to handle long-context inputs more
// effectively. The following properties can determine how NPUW behaves.
//
// NPUW_LLM_PREFILL_CHUNK_SIZE:
// Controls the chunk size for prompt prefill, which determines the granularity of input dynamism.
// Default value: 1024.
//
// NPUW_F16IC:
// Forces subgraph interconnect tensors to use FP16 precision when they would otherwise be FP32, provided that the
// partitioning pipeline is enabled. Setting this property to False may improve accuracy.
// Default value: True
//

using EmbeddingResult = std::variant<std::vector<float>, std::vector<int8_t>, std::vector<uint8_t>>;
using EmbeddingResults =
std::variant<std::vector<std::vector<float>>, std::vector<std::vector<int8_t>>, std::vector<std::vector<uint8_t>>>;
Expand Down
57 changes: 57 additions & 0 deletions src/cpp/src/rag/npu/text_embedding_pipeline.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
// Copyright (C) 2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0

#include "text_embedding_pipeline.hpp"

#include "openvino/core/except.hpp"
#include "rag/text_embedding_utils.hpp"
#include "utils.hpp"

namespace ov {
namespace genai {

InferRequest create_text_embedding_npu_request(std::shared_ptr<ov::Model>& model,
const TextEmbeddingPipeline::Config& config,
const ov::AnyMap& properties,
std::optional<size_t> max_position_embeddings,
const bool is_seq_len_fixed) {
if (config.batch_size.has_value() && is_seq_len_fixed) {
utils::reshape_model(model, config, max_position_embeddings);
}

ov::CompiledModel compiled_model;
if (model->is_dynamic()) {
bool is_padding_on_left = config.padding_side.has_value() && config.padding_side.value() == "left";
if (is_padding_on_left && is_seq_len_fixed && config.pooling_type != TextEmbeddingPipeline::PoolingType::MEAN) {
OPENVINO_THROW("Padding on left is only supported for the MEAN pooling type for dynamic inputs models."
" In order to fix model shape, set batch_size, max_length and pad_to_max_length in the "
"configuration.");
}

auto kv_pos = utils::get_kv_axes_pos(model);
utils::KVDesc kv_desc;
std::tie(compiled_model, kv_desc) =
utils::compile_decoder_for_npu_text_embedding(model, properties, kv_pos, config);
} else {
ov::Core core = utils::singleton_core();
model = utils::apply_postprocessing(model, config);
compiled_model = core.compile_model(model, "NPU", properties);
}
utils::print_compiled_model_properties(compiled_model, "npu text embedding model");
return compiled_model.create_infer_request();
}

InferRequest create_text_embedding_npu_post_request(std::shared_ptr<ov::Model>& model,
const TextEmbeddingPipeline::Config& config) {
if (model->is_dynamic()) {
ov::Core core = utils::singleton_core();
auto post_model = utils::create_post_model(model, config);
auto post_compiled_model = core.compile_model(post_model, "CPU");
return post_compiled_model.create_infer_request();
} else {
return InferRequest{};
}
}

} // namespace genai
} // namespace ov
22 changes: 22 additions & 0 deletions src/cpp/src/rag/npu/text_embedding_pipeline.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
// Copyright (C) 2026 Intel Corporation
// SPDX-License-Identifier: Apache-2.0

#pragma once

#include "openvino/genai/rag/text_embedding_pipeline.hpp"
#include "openvino/runtime/infer_request.hpp"

namespace ov {
namespace genai {

InferRequest create_text_embedding_npu_request(std::shared_ptr<ov::Model>& model,
const TextEmbeddingPipeline::Config& config,
const ov::AnyMap& properties,
std::optional<size_t> max_position_embeddings,
const bool is_seq_len_fixed);

InferRequest create_text_embedding_npu_post_request(std::shared_ptr<ov::Model>& model,
const TextEmbeddingPipeline::Config& config);

} // namespace genai
} // namespace ov
Loading
Loading