Update gcp gpu image #6417

zpoint · 2025-07-29T09:07:21Z

Follow up on #6191

Build new gcp's CPU and GPU image.

Above PR updates the catalog repo, which is required to make it work.

The failure also fail on master, not related to this PR.

Old base image doesn't work without upgrading GCC

Tried to revert the base image to ubuntu-2204-jammy-v20240927 but failed @Michaelvll :

Line 859-860:
What happened:

The base image ubuntu-2204-jammy-v20240927 started with an older kernel

During the provisioning process, something installed the linux-gcp package

This automatically pulled in the latest GCP kernel 6.8.0-1015-gcp which was compiled with GCC 12

The NVIDIA driver then tried to compile against this newer kernel and failed

The root cause is that GCP's guest environment automatically installs the latest linux-gcp kernel during system initialization, and this kernel was compiled with GCC 12. The hardcoded base image approach won't work because the kernel gets updated during the provisioning process.

Before this PR:

(sky) ➜  skypilot git:(master) sky launch --infra gcp 'ls /home' 
Command to run: ls /home
Considered resources (1 node):
------------------------------------------------------------------------------------
 INFRA                 INSTANCE        vCPUs   Mem(GB)   GPUS   COST ($)   CHOSEN   
------------------------------------------------------------------------------------
 GCP (us-central1-a)   n4-standard-8   8       32        -      0.38          ✔     
------------------------------------------------------------------------------------
Launching a new cluster 'sky-f306-zepingguo'. Proceed? [Y/n]: Y
⚙︎ Launching on GCP us-central1 (us-central1-a).
└── Instance is up.
✓ Cluster launched: sky-f306-zepingguo.  View logs: sky api logs -l sky-2025-07-24-22-31-20-663541/provision.log
⚙︎ Syncing files.
⚙︎ Job submitted, ID: 1
├── Waiting for task resources on 1 node.
└── Job started. Streaming logs... (Ctrl-C to exit log streaming; job will not be killed)
(sky-cmd, pid=3143) adithyansujithkumar
(sky-cmd, pid=3143) cooperc
(sky-cmd, pid=3143) danz
(sky-cmd, pid=3143) gcpuser
(sky-cmd, pid=3143) seungjinyang
(sky-cmd, pid=3143) ubuntu
(sky-cmd, pid=3143) yikaluo
✓ Job finished (status: SUCCEEDED).

After updated catalog to new image:

(sky) ➜  skypilot git:(master) sky launch --infra gcp 'ls /home'
Command to run: ls /home
Considered resources (1 node):
------------------------------------------------------------------------------------
 INFRA                 INSTANCE        vCPUs   Mem(GB)   GPUS   COST ($)   CHOSEN   
------------------------------------------------------------------------------------
 GCP (us-central1-a)   n4-standard-8   8       32        -      0.38          ✔     
------------------------------------------------------------------------------------
Launching a new cluster 'sky-e6da-zepingguo'. Proceed? [Y/n]: Y
⚙︎ Launching on GCP us-central1 (us-central1-a).
└── Instance is up.
✓ Cluster launched: sky-e6da-zepingguo.  View logs: sky api logs -l sky-2025-07-24-22-47-00-597721/provision.log
⚙︎ Syncing files.
⚙︎ Job submitted, ID: 1
├── Waiting for task resources on 1 node.
└── Job started. Streaming logs... (Ctrl-C to exit log streaming; job will not be killed)
(sky-cmd, pid=2988) gcpuser
(sky-cmd, pid=2988) ubuntu
✓ Job finished (status: SUCCEEDED).

Tested (run the relevant ones):

All smoke tests: /smoke-test --gcp (CI) or pytest tests/test_smoke.py (local)

zpoint · 2025-07-30T07:26:28Z

/smoke-test --gcp

zpoint · 2025-07-30T08:19:58Z

/smoke-test --gcp -k test_minimal

zpoint · 2025-07-30T08:31:53Z

sky/catalog/gcp_catalog.py

        _image_df = common.read_catalog('gcp/images.csv',
                                        pull_frequency_hours=0)
        image_id = common.get_image_id_from_tag_impl(_image_df, tag, region)
+    if tag == 'skypilot:custom-cpu-ubuntu-2204':


These test lines were added for testing and will be removed after the test passes before the PR is merged.

zpoint · 2025-07-30T08:47:35Z

/smoke-test --gcp

zpoint · 2025-07-31T03:26:31Z

The failure also fail on master, not related to this PR.

zpoint · 2025-07-31T03:29:01Z

All tests pass. Could u help take a look on this PR and the catalog change? @Michaelvll

zpoint · 2025-07-31T03:29:30Z

Or @aylei Could u help take a look?

zpoint · 2025-08-13T05:10:18Z

/smoke-test --gcp

Michaelvll

Thanks @zpoint! Could you help run a few recent LLM examples we have to make sure the cuda driver does not break any modern frameworks (may need to update the versions of those framework in the examples to the latest). Also, we should include the tests for those framework in our smoke tests as well.:
https://github.com/skypilot-org/skypilot/tree/master/llm/deepseek-r1-distilled
https://github.com/skypilot-org/skypilot/blob/master/llm/sglang/llama2.yaml

zpoint · 2025-08-13T08:36:03Z

/smoke-test --gcp -k test_deepseek_r1_vllm

zpoint · 2025-08-13T08:36:08Z

/smoke-test --gcp -k test_sglang_llama2_serving

zpoint · 2025-08-13T09:24:46Z

/smoke-test --gcp -k test_deepseek_r1_vllm
/smoke-test --gcp -k test_sglang_llava_serving

zpoint · 2025-08-13T09:25:39Z

There're so many test cases in example, added 2 cases. Create an issue as follow up to add rest.

zpoint · 2025-08-13T09:42:22Z

/smoke-test --gcp -k test_deepseek_r1_vllm
/smoke-test --gcp -k test_sglang_llava_serving

zpoint · 2025-08-13T09:47:03Z

/smoke-test --aws -k test_deepseek_r1_vllm
/smoke-test --aws -k test_sglang_llava_serving

zpoint · 2025-08-13T09:47:12Z

/smoke-test --azure -k test_deepseek_r1_vllm
/smoke-test --azure -k test_sglang_llava_serving

zpoint · 2025-08-13T11:26:31Z

/smoke-test --azure -k test_deepseek_r1_vllm

zpoint · 2025-08-13T13:56:33Z

/smoke-test --gcp -k test_deepseek_r1_vllm
/smoke-test --gcp -k test_sglang_llava_serving

zpoint · 2025-08-13T15:33:38Z

/smoke-test --gcp

zpoint · 2025-08-14T00:36:46Z

/smoke-test --gcp

zpoint · 2025-08-14T05:44:03Z

The failure tests re not related, see #6669

zpoint · 2025-08-14T06:21:22Z

@Michaelvll Can we proceed with the review && merge? The GCP test result looks good.

Michaelvll · 2025-08-20T03:20:08Z

cc'ing @cg505 for a final look

Michaelvll

Sorry for the delay @zpoint! Thanks for adding this! Please find the comments below : )

tests/smoke_tests/test_examples.py

Michaelvll · 2025-09-23T01:41:47Z

sky/catalog/images/provisioners/cuda.sh


-# Download architecture-specific CUDA keyring package
+# CRITICAL FIX: Install GCC 12 for kernel 6.8+ compatibility
+# GCP's kernel 6.8.0-1033-gcp was built with GCC 12, but NVIDIA drivers expected GCC 11


I am a bit confused by the commands, if nvidia driver expects GCC 11, why we are setting GCC 12 as the default compiler to use? Also, can we just use an older GCP base ubuntu image instead, e.g. just use the version that was used for building the previous images? Manually installing GCC needs extra care.

I think the comments were written when testing several different GCC versions, and only GCC12 works and is kept.

I'm trying to reuse the cuda.sh to make as few changes as possible now.

Michaelvll · 2025-09-23T01:45:54Z

sky/catalog/images/provisioners/cuda.sh

+echo "=== Setting up CUDA environment ==="
+# Add CUDA to system-wide profile
+echo 'export PATH="/usr/local/cuda/bin:$PATH"' | sudo tee -a /etc/profile
+echo 'export LD_LIBRARY_PATH="/usr/local/cuda/lib64:$LD_LIBRARY_PATH"' | sudo tee -a /etc/profile


Why this was not needed previously?

Worth adding /sbin in the path as well? https://github.com/skypilot-org/skypilot/blob/master/llm/qwen/qwen3-235b.yaml#L30

I think that's also because of the custom CUDA and GCC version installation.

zpoint · 2025-09-23T14:30:26Z

We need the catalog from skypilot-org/skypilot-catalog#150 to be merged and updated so CI can pass.

zpoint · 2025-09-23T14:56:25Z

/smoke-test --gcp

gcp build

0a8eb0b

zpoint requested a review from Michaelvll July 29, 2025 11:06

zpoint added 3 commits July 30, 2025 11:11

Merge branch 'master' into dev/zeping/gcp_images

d3fe9cb

fix script

b3215d5

test

0ce7921

fix failure

32c5ec9

zpoint commented Jul 30, 2025

View reviewed changes

zpoint mentioned this pull request Jul 30, 2025

Update GCP image to clean up user dir skypilot-org/skypilot-catalog#138

Merged

zpoint requested a review from aylei July 31, 2025 03:28

specific version

ce588af

Michaelvll reviewed Aug 13, 2025

View reviewed changes

zpoint added 3 commits August 13, 2025 15:00

Merge branch 'master' into dev/zeping/gcp_images

6e88ae2

sky test llm

5be79ad

comment

d98adcf

zpoint added 2 commits August 13, 2025 16:53

fix test failure

c7f6238

fix failure

37651ff

zpoint mentioned this pull request Aug 13, 2025

Add all llm examples in smoke tests #6651

Closed

longer timeout

bc60ca6

longer timeout

8deb70e

fix test

d0836b3

fix test

948239c

revert catalog debug change

ccc9838

zpoint requested a review from Michaelvll August 14, 2025 06:19

Michaelvll requested a review from cg505 August 20, 2025 03:20

Michaelvll reviewed Sep 23, 2025

View reviewed changes

zpoint added 3 commits September 23, 2025 16:09

merge master

7f5173c

renmae to test_examples

3bc95e2

restore cuda change

9272b2d

zpoint mentioned this pull request Sep 23, 2025

Update GCP image to clean up user dir skypilot-org/skypilot-catalog#150

Open

resolve comment and rebuild

87a5f72

zpoint requested a review from Michaelvll September 23, 2025 14:30

zpoint added 2 commits September 23, 2025 22:37

debug

53c18ef

comment

b12166c

zpoint force-pushed the dev/zeping/gcp_images branch from d2d3ece to b12166c Compare September 24, 2025 04:26

Update gcp gpu image #6417

Are you sure you want to change the base?

Update gcp gpu image #6417

Uh oh!

Conversation

zpoint commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Old base image doesn't work without upgrading GCC

Before this PR:

After updated catalog to new image:

Uh oh!

zpoint commented Jul 30, 2025

Uh oh!

zpoint commented Jul 30, 2025

Uh oh!

zpoint Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

zpoint commented Jul 30, 2025

Uh oh!

zpoint commented Jul 31, 2025

Uh oh!

zpoint commented Jul 31, 2025

Uh oh!

zpoint commented Jul 31, 2025

Uh oh!

zpoint commented Aug 13, 2025

Uh oh!

Michaelvll left a comment

Choose a reason for hiding this comment

Uh oh!

zpoint commented Aug 13, 2025

Uh oh!

zpoint commented Aug 13, 2025

Uh oh!

zpoint commented Aug 13, 2025

Uh oh!

zpoint commented Aug 13, 2025

Uh oh!

zpoint commented Aug 13, 2025

Uh oh!

zpoint commented Aug 13, 2025

Uh oh!

zpoint commented Aug 13, 2025

Uh oh!

zpoint commented Aug 13, 2025

Uh oh!

zpoint commented Aug 13, 2025

Uh oh!

zpoint commented Aug 13, 2025

Uh oh!

zpoint commented Aug 14, 2025

Uh oh!

zpoint commented Aug 14, 2025

Uh oh!

zpoint commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Michaelvll commented Aug 20, 2025

Uh oh!

Michaelvll left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Michaelvll Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

zpoint Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michaelvll Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

Michaelvll Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

zpoint Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

zpoint commented Jul 29, 2025 •

edited

Loading

zpoint commented Aug 14, 2025 •

edited

Loading

zpoint Sep 23, 2025 •

edited

Loading