Jenkins plugins not getting upgraded through helm chart

[ssolanki38]

Describe the bug

We have deployed Jenkins in a Kubernetes cluster using the Jenkins helm chart. When we want to upgrade the LTS version of a Jenkins instance, we also want to upgrade the plugins. As part of the helm upgrade, we pass in a plugins.txt file with the list of plugins and plugin versions we would like to upgrade to. We also specify the jenkins LTS version to upgrade to.
Then we execute the helm upgrade command.

The problem we're seeing is that sometimes the plugins do not get upgraded while the LTS version does. This does not happen on all the instances we have, only some. It is not clear why the plugins remain at the same version. In the init container, we see the logs of the new plugins getting downloaded and copied to the shared directory /var/jenkins_plugins. Is there a recommended way to upgrade the plugins through the helm chart? It would also be helpful to know if there's something we can do to debug this situation. We are not seeing any helpful logs in the jenkins-master container. Thanks in advance.

Version of Helm and Kubernetes

- Helm: version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}
- Kubernetes: Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8", GitCommit:"7061dbbf75f9f82e8ab21f9be7e8ffcaae8e0d44", GitTreeState:"clean", BuildDate:"2022-03-16T14:04:34Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Chart version

Our chart is from a few years ago. I'm not sure of the exact version.

What happened?

1. Deploy an instance with LTS version x, and some plugins at specific versions
2. Supply a newer LTS version and plugins.txt with newer versions in the values file. 
3. Run helm upgrade command with this values file.

What you expected to happen?

The newly downloaded jenkins plugins should be loaded into the Jenkins instance, but they remain using the previous versions.

How to reproduce it

No response

Anything else we need to know?

No response

[sanketbisne]

Even I am facing the same issue

[shezaan-hashgraph]

Same here

[brandonp42]

Have any of you tried adding this to your helm values?

controller:
  containerEnv:
    - name: PLUGINS_FORCE_UPGRADE
      value: "true"
    - name: TRY_UPGRADE_IF_NO_MARKER
      value: "true"

[nkazarian-spokeo]

I noticed this as well. The script that the initContainers runs has this line:

# Copy plugins to shared volume
yes n | cp -i {{ .Values.controller.jenkinsRef }}/plugins/* /var/jenkins_plugins/;

I see multiple issues with this: /var/jenkins_plugins is not even a directory that exists. If you try this command manually, it will fail. Also...assuming you create that directory, the yes n is suspicious because then it would be inputting n to each cp -i. I tried manually moving one of my updated plugins to /var/jenkins_home/plugins and it worked. I think there's a real issue here.

[ssolanki38]

@nkazarian-spokeo Correct. I was confused about the same thing. I also had to manually run extra commands to copy the upgraded plugins to the JENKINS_HOME plugins directory.

[brandonp42]

@nkazarian-spokeo @ssolanki38

Which script are you guys talking about and running in which container? /var/jenkins_plugins does exist on the init container mounted from the volume plugin-dir. Then that volume gets mounted to /usr/share/jenkins/ref/plugins on the main jenkins container.

Ie there's a multi-step process going on here to prepare the plugins for consumption by the final jenkins container.

[brandonp42]

@nkazarian-spokeo @ssolanki38

In the main Jenkins container at startup it runs this (https://github.com/jenkinsci/docker/blob/master/jenkins-support#L42) to copy plugins. If you follow the logic there you will see why I suggested adding the environment variables in comment #1115 (comment).

[nkazarian-spokeo]

Then maybe something with the yes n not playing nicely with plugins that have their versions bumped but retain the same file name and .jpi extension? Since the problem seems unique to upgrading existing packages.

[brandonp42]

The second script (in the main Jenkins container) refuses to upgrade extensions if you ever install or upgrade them from the web user interface. That’s the default “as designed” behavior. But if you set the environment variables from my prior comment then it will go ahead and upgrade them anyway.

If you check the ${JENKINS_HOME}/copy_reference_file.log you should see messages saying it skipped your plugin upgrades.

I think the real issue is this behavior needs to be better documented. It certainly took me a while to figure this out on my production system.

[timja]

Correct if you ever upgrade them in the Web UI they won't be upgraded,

Its documented here:

https://github.com/jenkinsci/docker#upgrading-plugins

By default, plugins will be upgraded if they haven't been upgraded manually and if the version from the docker image is newer than the version in the container. Versions installed by the docker image are tracked through a marker file.

[brandonp42]

@timja I think it would be helpful if there was some commentary about it in the Helm chart documentation, at least pointing to the link you gave and an example of setting the environment variables in the Helm values. Would you be open to either adding that or accepting a pull request?

[timja]

Happy to accept a pull request.