operator loops endlessly on patching webhooks due to random order of namespace values

[davidkarlsen]

we see this endless rate of patching webhooks, which overwhelms the api-server

I0508 13:08:12.369462       1 namespacescope_controller.go:1272] Patching webhook scope for: moperandrequest.kb.io-p2mzp
I0508 13:08:12.409561       1 namespacescope_controller.go:1282] Patching validatingwebhookconfig scope for: vcommonservice.kb.io-mk5ck, control by csv ibm-common-service-operator.v4.6.13/ibm-common-services
I0508 13:08:12.409717       1 namespacescope_controller.go:1312] Patching webhookconfig scope for: vcommonservice.kb.io-mk5ck
I0508 13:08:12.428419       1 namespacescope_controller.go:1035] Reconciling NamespaceScope: ibm-common-services/common-service for patching operator CSV
I0508 13:08:12.429564       1 namespacescope_controller.go:1086] Patching webhookconfiguration for CSV ibm-common-service-operator.v4.6.13
I0508 13:08:12.429686       1 namespacescope_controller.go:1242] Patching mutatingwebhookconfig scope for: moperandrequest.kb.io-p2mzp, control by csv ibm-common-service-operator.v4.6.13/ibm-common-services
I0508 13:08:12.429739       1 namespacescope_controller.go:1272] Patching webhook scope for: moperandrequest.kb.io-p2mzp
I0508 13:08:12.451072       1 namespacescope_controller.go:1282] Patching validatingwebhookconfig scope for: vcommonservice.kb.io-mk5ck, control by csv ibm-common-service-operator.v4.6.13/ibm-common-services
I0508 13:08:12.451095       1 namespacescope_controller.go:1312] Patching webhookconfig scope for: vcommonservice.kb.io-mk5ck
I0508 13:08:12.471872       1 namespacescope_controller.go:1035] Reconciling NamespaceScope: ibm-common-services/common-service for patching operator CSV
I0508 13:08:12.472786       1 namespacescope_controller.go:1086] Patching webhookconfiguration for CSV ibm-common-service-operator.v4.6.13
I0508 13:08:12.472861       1 namespacescope_controller.go:1242] Patching mutatingwebhookconfig scope for: moperandrequest.kb.io-p2mzp, control by csv ibm-common-service-operator.v4.6.13/ibm-common-services

and sure enough, if I
k get ValidatingWebhookConfiguration vcommonservice.kb.io-mk5ck -o yaml --watch
I see the resource being updated all the time, and the values in:

namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: In
      values:

are never in the same order.

The list contains 29 values - but it seems the operator does not put these values in the same order - so the reconciliation never settles and will put high strain on api-server as well as etc (many writes a second).
Same symptom on the mutating one: moperandrequest.kb.io-p2mzp

[davidkarlsen]

@bitscuit perhaps sorting the values should fix the issue?

[Daniel-Fan]

Thank you @davidkarlsen for raising the issue!

We have opened an issue internally to track the fix for this.

[davidkarlsen]

@Daniel-Fan when will a release be available?