GPU Support?

[jordanparker6]

Does qdrant support running on GPUs with CUDA?

[generall]

Hi, there is no such feature in the current version. We are considering using GPUs, but we don't make it a priority. In our experience, a much larger gain can be achieved by distributing the service between regular CPU machines. This would be cheaper and more efficient

[jordanparker6]

Thanks for the response. I really like this project. I will take your advice and try it out in a K8 cluster.

[dingusagar]

@generall , are there any guidelines on how to set up qdrant in a cluster setup. Will installing separate instances of qdrant on multiple nodes and mounting an AWS EFS on all the nodes to store the qdrant directory (collections and indices files) do the job. Or is there anything else that needs to be taken care of?

[generall]

@generall , are there any guidelines on how to set up qdrant in a cluster setup. Will installing separate instances of qdrant on multiple nodes and mounting an AWS EFS on all the nodes to store the qdrant directory (collections and indices files) do the job. Or is there anything else that needs to be taken care of?

Hi @dingusagar, cluster deployment is not yet supported. We are working on it as a part of v1.0 release. Please check out our roadmap for more details - https://github.com/qdrant/qdrant/blob/master/docs/roadmap/README.md

As a temporary workaround, you can try to implement your own proxy which will route requests among independent instances.

[kinthaiofficial]

GPU support for vector search is a meaningful optimization at scale — the gap between CPU and GPU HNSW search grows as collection size and dimensionality increase.

Current state of GPU vector search (as of 2024-2025):

• FAISS: GPU support for flat (brute-force) and IVF indexes via cuVS/FAISS-GPU. Excellent for batch queries where you can saturate the GPU.
• Milvus: GPU index building and search via NVIDIA cuVS integration. IVF_FLAT_GPU, CAGRA index types.
• Weaviate: No native GPU; relies on CPU HNSW.
• Qdrant: CPU HNSW currently; GPU support has been discussed but not shipped.

When GPU actually helps:

CPU HNSW: optimized for latency on single/few queries
           ~1-5ms per query, scales poorly with parallelism

GPU flat: optimized for throughput on many concurrent queries
           GPU memory bandwidth dominates (~10-50ms but 10k queries at once)

For RAG workloads (many queries from many users), the throughput advantage of GPU search matters. For single-user interactive search, CPU HNSW's latency advantage often wins.

The practical alternative: If you need more throughput than CPU HNSW provides, horizontal scaling (more Qdrant nodes) is often more operationally straightforward than GPU setup. GPU vector search requires careful batching to amortize transfer overhead.

Is the use case throughput (many concurrent queries) or query latency optimization?

[timvisee]

Since Qdrant 1.13 and onward GPU usage is supported. Documentation

Note that it uses GPU for rapid HNSW graph construction. Search will still use CPU. Actual vector search on GPU is unreasonable since it's extremely expensive, it would require the whole HNSW graph and its vectors to be inside video memory which has a severe cost.