Using CUDA

[ItsMeJacky]

How to run faster-whisper using CUDA ?

[rufinus]

cudnn libs + --device cuda

on my LXC setup i followed this article https://sluijsjes.nl/2024/05/18/coral-and-nvidia-passthrough-for-proxmox-lxc-to-install-frigate-video-surveillance-server/

you just have to include the correct libs in the container, or map damn from the host to the container.

i ended up building the cointainer myself with this dockerfile:

FROM rhasspy/wyoming-whisper:2.1.0
RUN apt update && apt install -y wget software-properties-common
RUN wget https://developer.download.nvidia.com/compute/cudnn/8.9.7.29/local_installers/12.2/cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN dpkg -i cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN cp /var/cudnn-local-repo-debian11-8.9.7.29/cudnn-*-keyring.gpg /usr/share/keyrings/
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.1-1_all.deb
RUN dpkg -i cuda-keyring_1.1-1_all.deb
RUN add-apt-repository contrib && apt --allow-releaseinfo-change update
RUN apt -y install libcudnn8 libcublas-12-0
RUN rm cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb

took some time to find the right versions to make it run.

[ssamjh]

👍 For official GPU support, would be nice to speed things up :)

[ssamjh]

👍 for this feature. I already have a GPU in my server for other applications :)

[Mithras]

Also, is it possible to make it unload model based on some --keep-alive param?

[alienatedsec]

How to run faster-whisper using CUDA ?

Simple and works out of the box - https://github.com/linuxserver/docker-faster-whisper

[AnkushMalaker]

I made PR #44 to allow building it directly from this repository, and use CUDA.
It works perfectly with just docker compose up -d or using the build scripts in the README.
@synesthesiam will need to upload a gpu based image to rhasspy dockerhub if they want to allow people to pull without building locally.

[danieljkemp]

RUN apt update && apt install -y wget software-properties-common \
  && wget https://developer.download.nvidia.com/compute/cudnn/9.1.0/local_installers/cudnn-local-repo-debian11-9.1.0_1.0-1_amd64.deb \
  && dpkg -i cudnn-local-repo-debian11-9.1.0_1.0-1_amd64.deb \
  && cp /var/cudnn-local-repo-debian11-9.1.0/cudnn-*-keyring.gpg /usr/share/keyrings/ \
  && wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.1-1_all.deb \
  && dpkg -i cuda-keyring_1.1-1_all.deb \
  && add-apt-repository contrib && apt --allow-releaseinfo-change update \
  && apt -y install libcudnn9-cuda-12 libcublas-12-0 \
  && rm cudnn-local-repo-debian11-9.1.0_1.0-1_amd64.deb

working with 2.4.0

[xiasi0]

Using CUDNN9.6 and 2.4.0 versions does not work properly. I hope someone can give me a hint

Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so}

Invalid handle. Cannot load symbol cudnnCreateTensorDescriptor

Do I have to use CUDNN9.1？

#35 (comment)
It can now work using CUDA imag

[jjsmackay]

Thanks to @rufinus, I got this working on a GTX 970M.

Sadly this poxy little GPU only supports CUDA Compute Capability 5.2, and I couldn't get it working on CUDA 12. The CUDA 11 workaround discussed here worked a treat.

Getting <1s translation using small.en model and beam-size 3 with just 3GB VRAM. Happy days!

FROM rhasspy/wyoming-whisper:2.1.0
RUN apt update && apt install -y wget software-properties-common
RUN wget https://developer.download.nvidia.com/compute/cudnn/8.9.7.29/local_installers/11.8/cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN dpkg -i cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN cp /var/cudnn-local-repo-debian11-8.9.7.29/cudnn-*-keyring.gpg /usr/share/keyrings/
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.1-1_all.deb
RUN dpkg -i cuda-keyring_1.1-1_all.deb
RUN add-apt-repository contrib && add-apt-repository non-free && apt --allow-releaseinfo-change update
RUN apt -y install libcudnn8 libcublas11
RUN rm cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN pip install --force-reinstall ctranslate2==3.24.0