Add RayJob training example using pytorch resnet image classifier #2107
Conversation
andrewsykim
force-pushed
the
pytorch-lightning-image-classifier
branch
from
24b91ee to
da09f40
Compare
andrewsykim
force-pushed
the
pytorch-lightning-image-classifier
branch
from
da09f40 to
39c73c9
Compare
| bucketName: GCS_BUCKET | ||
| mountOptions: "implicit-dirs,uid=1000,gid=100" | ||
| workerGroupSpecs: | ||
| - replicas: 4 |
There was a problem hiding this comment.
When I run kubectl create -f ..., I got the following error. We should specify minReplicas and maxReplicas here.
error: error validating "ray-job.pytorch-image-classifier.yaml": error validating data: [ValidationError(RayJob.spec.rayClusterSpec.workerGroupSpecs[0]): missing required field "maxReplicas" in io.ray.v1.RayJob.spec.rayClusterSpec.workerGroupSpecs, ValidationError(RayJob.spec.rayClusterSpec.workerGroupSpecs[0]): missing required field "minReplicas" in io.ray.v1.RayJob.spec.rayClusterSpec.workerGroupSpecs]; if you choose to ignore these errors, turn validation off with --validate=false
There was a problem hiding this comment.
Huh, that's weird -- I didn't run into that error, maybe I didn't have validation enabled? I added both
There was a problem hiding this comment.
Interesting. Both of them are required.
kuberay/ray-operator/apis/ray/v1/raycluster_types.go
Lines 53 to 58 in 7fb46ab
There was a problem hiding this comment.
Those have kubebuilder defaults though, should it error if not specified?
There was a problem hiding this comment.
In my understanding, the kubebuilder defaults will never be used because there is no omitempty in either MinReplicas or MaxReplicas.
MaxReplicas *int32 `json:"maxReplicas,omitempty"`| containers: | ||
| - name: ray-worker | ||
| image: rayproject/ray-ml:2.9.0-gpu | ||
| lifecycle: |
There was a problem hiding this comment.
We can remove this.
lifecycle:
preStop:
exec:
command: [ "/bin/sh","-c","ray stop" ]
…ckpointing Signed-off-by: Andrew Sy Kim <[email protected]>
Signed-off-by: Andrew Sy Kim <[email protected]>
Signed-off-by: Andrew Sy Kim <[email protected]>
Signed-off-by: Andrew Sy Kim <[email protected]>
Signed-off-by: Andrew Sy Kim <[email protected]>
Signed-off-by: Andrew Sy Kim <[email protected]>
andrewsykim
force-pushed
the
pytorch-lightning-image-classifier
branch
from
39c73c9 to
7b9759d
Compare
Signed-off-by: Andrew Sy Kim <[email protected]>
I removed changes in the pytorch-text-classifier sample. I wanted to have an example of it where it uses a shared filesystem, but I guess we can keep those two examples separate |
| @@ -131,15 +131,16 @@ def train_func(config): | |||
| # The checkpoints and metrics are reported by `RayTrainReportCallback` | |||
| run_config = RunConfig( | |||
| name="ptl-sent-classification", | |||
| storage_path="/mnt/cluster_storage", | |||
There was a problem hiding this comment.
Why do we need this change? This change may cause users fail to follow this doc.
There was a problem hiding this comment.
sorry, this was leftover from the last change, reverting it
Signed-off-by: Andrew Sy Kim <[email protected]>
| bucketName: GCS_BUCKET | ||
| mountOptions: "implicit-dirs,uid=1000,gid=100" | ||
| workerGroupSpecs: | ||
| - replicas: 4 |
There was a problem hiding this comment.
In my understanding, the kubebuilder defaults will never be used because there is no omitempty in either MinReplicas or MaxReplicas.
MaxReplicas *int32 `json:"maxReplicas,omitempty"`
|
The CI failure is not related to this PR. Merge. |
Why are these changes needed?
Add an example RayJob based on the Finetuning a Pytorch Image Classifier with Ray Train example.
This example will be referenced for user guides that demonstrate distributed checkpointing with GCSFuse.
Also updates the existing pytorch text classifier with an example using GCSFuse.
Related issue number
Checks