remove the rolling update triggering when upgrading the Cluster Operator

[282ori]

Hi,
Today when we upgrade the Cluster Operator it causes rolling update for ALL the Kafka Clusters that the operator manages.
I understand that its for CVE fixes and some other reasons.
We know that rolling updates have effect on the Kafka Cluster.
For example, If the pod rolled was the leader partition for a topic a new leader is required and the election takes some time
and in that time the produce CAN'T send message to that partition.
I've seen that @scholzj said: "I'm not sure if I mentioned it before in this thread or somewhere else. But the rolling updates in general have always some disruption."
I find this issue very problematic, as the Operator forces rolling update on all the clusters in the same time.
I propose allowing MANUAL rolling update for the Kafka Clusters (I saw these were proposals for the mirrorMaker and the Connect).
Could these changes occur dynamically like we can change configurations of kafka (spec.config) without rolling update?
Or maybe other solutions could be found.

[scholzj]

If you want, you can for example pause the reconciliation of the Kafka clusters before you do the operator upgrade and then unpause them in any order and timing you want. That essentially does your manual rolling update. But the rolling update is not just because of the CVEs - that is only one of the aspects. It is also needed for the operator to understand the state and configuration of the clusters it is supposed to operate. So I do not think that the rolling update can be really avoided in general.

[282ori]

1. If I pause the reconcilation of the Kafka Cluster, when I upgrade the Cluster Operator would it still manage this Kafka cluster?
   Because if it will be unsupported, if one pod fails the operator won't create it again (as part of the SPS) and its still a problem.
2. Can't the state and cofigurations be updated dynamically on the pods? Exactly as you can change configurations like 'message.max.bytes' (its update in the custom-config dir)?

[scholzj]

Re 1) It will not. But it cannot do that anyway without rolling the pods.
Re 2) It can likely change message.max.bytes. But it might not be able to do many other things and changes.

[scholzj]

if one pod fails the operator won't create it again (as part of the SPS) and its still a problem.

Just to add -> it should restart the pod even when the Kafka CR is paused. But it would not roll it if needed, it will not renew the certificates if they expire while it is paused or change their SANs just because something in the cluster changed etc.

[282ori]

Is it possible to open a feture request about this issue?
The rolling update might be a deal breaker for those who can not tolerate downtime.
It might be helpful to examine if its possible to upgrade the operator without downtime like in other kubernetes operators...

[scholzj]

You can always open a new issue. But I guess you should check if there isn't an existing feature request. Also, keep in mind that it was discussed many times in the past and the outcome was always the same -> it is not possible within some reasonable efforts. If you think it is super simple, easy, and obvious ... you should explain how to do it in a way that is easy and simple, as we apparently do not know it. Because otherwise it just wastes the time and the outcome will be exactly the same as before. I do not know what other operators are you referring to ... but in general, if you want to upgrade Strimzi you have to restart things to use the new binaries. That IMHO applies to all operators unless they do some weird patching of the binaries in memory. Keep in mind that Strimzi is also the operand image. There are agents for operationalizing Kafka, config providers to help with configuration etc. Different versions have also different config files etc. because Kafka is not a project that is done and does not develop anymore. There are many changes going on in every release.

One thing to keep in mind is that:

• Rolling update is not the same as downtime. There is no downtime when upgrading Strimzi assuming you properly configured your topics and clients!
• You control when you upgrade Strimzi so you control when the disruption happens.