feat: record last state transition times #2053
Merged
+157
−14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sydneyw-spotify
force-pushed
the
state-transitions
branch
from
f26cc9f to
9dc3145
Compare
|
@sydneyw-spotify just saw you update this PR recently. Feel free to ping me whenever this PR is ready for review. |
|
@kevin85421, thanks. This is ready for review now. Example input and output in the info gist link in PR description. |
davidxia
force-pushed
the
state-transitions
branch
from
ec886c7 to
4996fca
Compare
## Problem Statement My ML platform team runs the kuberay ray-operator. We want to measure the time it takes for RayCluster's to transition from their initial "unhealthy" state to some other state. This metric is important for us because our users want their RayClusters to start in a timely manner. It seems like neither the ray-operator nor RayClusters provide this info currently. ## Design Add a new `.status.stateTransitionTimes` field to the `RayCluster` custom resource. This field is a `map[ClusterState]*metav1.Time` that indicates the time of the last state transition for each state. This field is updated whenever the `.status.state` changes. * [original discussion doc](https://docs.google.com/document/d/14yPSZ9iLk7a0qEg14rNWr60Btz0HEeQ3oWKP-GN9QTM) * [related Slack thread](https://ray-distributed.slack.com/archives/C01CKH05XBN/p1709321264762029) * [example input and output RayClusters](https://gist.github.com/davidxia/205d2b23202356a2d3172c51e0912f35)
davidxia
force-pushed
the
state-transitions
branch
from
4996fca to
fdfb6c7
Compare
kevin85421
approved these changes
May 1, 2024
Thank you! |
4 tasks
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Problem Statement
My ML platform team runs the kuberay ray-operator. We want to measure the time
it takes for RayCluster's to transition from their initial "unhealthy" state to
some other state. This metric is important for us because our users want their
RayClusters to start in a timely manner. It seems like neither the ray-operator
nor RayClusters provide this info currently.
Design
Add a new
.status.stateTransitionTimesfield to theRayClustercustomresource. This field is a
map[ClusterState]*metav1.Timethat indicates thetime of the last state transition for each state. This field is updated
whenever the
.status.statechanges.manual testing steps:
make manifests generatekubectl config use-context CONTEXTmake docker-build docker-push deploy IMG=europe-west4-docker.pkg.dev/spotify-workbench/images/operator:$USER-$(git rev-parse --short=7 HEAD)kubectl --context CONTEXT apply -f /path/to/raycluster.yaml.status.stateTransitionTimesin the output ofkubectl --context CONTEXT -n NAMESPACE get rayclusters NAME -o yamlRelated issue number
Checks