Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ready check for cert-controller #2359

Closed
wants to merge 2 commits into from
Closed

Add ready check for cert-controller #2359

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2d60736
add check
forsaken628 Jun 16, 2024
a88647d
fix
forsaken628 Jul 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 18 additions & 4 deletions cmd/katib-controller/v1beta1/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ package main

import (
"flag"
"fmt"
"net/http"
"os"

"github.com/spf13/viper"
Expand Down Expand Up @@ -133,23 +135,35 @@ func main() {

ctx := signals.SetupSignalHandler()
certsReady := make(chan struct{})
defer close(certsReady)

// The setupControllers will register controllers to the manager
// after generated certs for the admission webhooks.
go setupControllers(mgr, certsReady, hookServer)

if initConfig.CertGeneratorConfig.Enable {
err := mgr.AddReadyzCheck("cert", func(_ *http.Request) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add this check to the controller readinessProbe ?
https://github.com/kubeflow/katib/blob/master/manifests/v1beta1/components/controller/controller.yaml#L41

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If not specified, readinessProbe checks all readiness checkers. the name refers to the name of the checker, not readinessEndpointName.

kubernetes-sigs/controller-runtime#1100

https://github.com/kubernetes-sigs/controller-runtime/blob/1ed345090869edc4bd94fe220386cb7fa5df745f/pkg/healthz/healthz.go#L147-L159

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@forsaken628 @tenzen-y Just for user visibility, maybe we should explicitly call out them in the controller Deployment ? It might be hard for user to check code to understand all probs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late reply, @forsaken628 what do you think about above comment ? Do we want to make that change and merge this PR ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the vast majority of case users only care about aggregation probes. We could add a comment somewhere.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just want to keep it consistent between Training Operator: https://github.com/kubeflow/training-operator/blob/master/manifests/base/deployment.yaml#L53 and Katib controller.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tenzen-y Any thoughts ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need to modify deployment.yaml. The individual check is not a new feature for controller-runtime, and usually users don't need to know it exists.

select {
case <-certsReady:
return nil
default:
return fmt.Errorf("certificates are not ready")
}
})
if err != nil {
log.Error(err, "Unable to add readyz check cert")
os.Exit(1)
}
if err = cert.AddToManager(mgr, initConfig.CertGeneratorConfig, certsReady); err != nil {
log.Error(err, "Failed to set up cert-generator")
os.Exit(1)
}
} else {
certsReady <- struct{}{}
close(certsReady)
}

log.Info("Setting up health checker.")
if err := mgr.AddReadyzCheck("readyz", hookServer.StartedChecker()); err != nil {
log.Error(err, "Unable to add readyz endpoint to the manager")
if err := mgr.AddReadyzCheck("webhook", hookServer.StartedChecker()); err != nil {
log.Error(err, "Unable to add readyz check webhook")
os.Exit(1)
}
if err = mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
Expand Down