distiluse-base-multilingual-cased-v2 error when start

[franklucky001]

System Info

image

text-embeddings-inference:turing-1.6-grpc

model id

sentence-transformers/distiluse-base-multilingual-cased-v2

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

error info

dense-embed  | 2025-04-23T03:14:03.297238Z  INFO text_embeddings_router: router/src/lib.rs:188: Maximum number of tokens per request: 128
dense-embed  | 2025-04-23T03:14:03.297495Z  INFO text_embeddings_core::tokenization: core/src/tokenization.rs:28: Starting 48 tokenization workers
dense-embed  | 2025-04-23T03:14:05.341111Z  INFO text_embeddings_router: router/src/lib.rs:230: Starting model backend
dense-embed  | 2025-04-23T03:14:05.918661Z  INFO text_embeddings_backend_candle: backends/candle/src/lib.rs:317: Starting DistilBertModel model on Cuda(CudaDevice(DeviceId(1)))
dense-embed  | 2025-04-23T03:14:07.818830Z ERROR text_embeddings_backend: backends/src/lib.rs:255: Could not start Candle backend: Could not start backend: cannot find tensor encoder.layer.0.attention.q_lin.weight
dense-embed  | Error: Could not create backend
dense-embed  | 
dense-embed  | Caused by:
dense-embed  |     Could not start backend: Could not start a suitable backend

Expected behavior

start service successful

[alvarobartt]

Hey @franklucky001 thanks for reporting! It seems that the keys used to pull the weights for the DistilBERT models were missing a clause AFAIK, I'll keep on investigating and provide a patch for Text Embeddings Inference (TEI), I'll let you know as soon as possible 🤗

[alvarobartt]

Here's the draft PR #614 in case you want to have a look at it, and use that branch in the meantime @franklucky001!

P.S. Also note that in the issue you mentioned the ghcr.io/huggingface/text-embeddings-inference:turing-1.6-grpc image, but the ghcr.io/huggingface/text-embeddings-inference:turing-1.7-grpc was recently released, it doesn't contain the aforementioned fix as it's still not merged, but flagging in any case!