New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Setup failure: no node has >= 1 GPU(s) in Nvidia Inference test #539

Merged

mattcjo merged 1 commit into aws:main from wwvela:inf

Jan 4, 2025

Contributor

wwvela commented Jan 4, 2025

Issue #, if available:

Nvidia Inference test randomly failed with error Setup failure: no node has >= 1 GPU(s). It is because after apply NVIDIA device plugin it will take some time for the gpu info propagated to the node status. And if the checkGpuCapacity happened before propagation finished, it will failed
To fix above issue, add retry for GPU capacity check


          add retry for gpu capacity check

f46bd8c

wwvela requested a review from mattcjo

January 4, 2025 12:41

mattcjo approved these changes

View reviewed changes

Contributor

mattcjo left a comment

LGTM

mattcjo merged commit 96fa4fc into aws:main

6 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet