Enable GPU Testing for LLM Blueprints

[andreyvelich]

What you would like to be added?

We will soon introduce LLM Blueprints which typically require GPUs to run them: #2410.
To support this, we need to explore using GitHub self-hosted runners with GPU support.

If we can get them, we need to see if we can deploy Kubernetes cluster with these runners and NVIDIA GPU Operator.

cc @Electronic-Waste @astefanutti @kubeflow/wg-training-leads @deepanker13 @saileshd1402 @franciscojavierarceo

Why is this needed?

We need GPU for our testing infrastructure.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[mahdikhashan]

@andreyvelich I did a very brief research and it seems like kind does not support GPUs natively, however there are custom configurations to enable it, one is a maintained fork from NVIDIA itself:

1. https://github.com/NVIDIA/nvkind

otherwise, minikube has native support:

2. https://minikube.sigs.k8s.io/docs/tutorials/nvidia/

[andreyvelich]

This is good news that minikube supports Nvidia device plugin!

I remember that @astefanutti and @franciscojavierarceo also explored how to leverage Nvidia with local k8s cluster: #2231 (comment)