Skip to content

Decoding of secrets into credentials through ${decodeBase64:${...}} doesn't work per the documentation #1370

New issue
New issue
Open
Open
Decoding of secrets into credentials through ${decodeBase64:${...}} doesn't work per the documentation#1370
Labels
bugSomething isn't working
@oliverlockwood

Description

@oliverlockwood
oliverlockwood
opened on May 7, 2025

Describe the bug

The additionalExistingSecrets variable templating (described in https://github.com/jenkinsci/helm-charts/blob/main/charts/jenkins/README.md#additional-secrets) does not seem to play nicely with the decodeBase64() functionality described in https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/docs/features/secrets.adoc#additional-variable-substitution.

I am using the following configuration:

controller:
  JCasC:
    configScripts:
      jenkins-casc-configs: |
        credentials:
          system:
            domainCredentials:
              - credentials:
                  - basicSSHUserPrivateKey:
                      id: "ec2-ssh-key-credential"
                      description: "jenkins Ed25519 SSH key, with username set to jenkins"
                      username: "jenkins"
                      usernameSecret: false
                      privateKeySource:
                        directEntry:
                          privateKey: ${decodeBase64:${jenkins-additional-secrets-ec2-worker-ssh-key-base64}}
                      scope: GLOBAL

The inner variable appears not to be interpreted; certainly, the resulting error looks like this:

java.lang.IllegalArgumentException: Illegal base64 character 24
	at java.base/java.util.Base64$Decoder.decode0(Unknown Source)
	at java.base/java.util.Base64$Decoder.decode(Unknown Source)
	at java.base/java.util.Base64$Decoder.decode(Unknown Source)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.SecretSourceResolver$DecodeBase64Lookup.lookup(SecretSourceResolver.java:204)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.FixedInterpolatorStringLookup.lookup(FixedInterpolatorStringLookup.java:272)
	at PluginClassLoader for commons-text-api//org.apache.commons.text.StringSubstitutor.resolveVariable(StringSubstitutor.java:1155)
	at PluginClassLoader for commons-text-api//org.apache.commons.text.StringSubstitutor.substitute(StringSubstitutor.java:1521)
	at PluginClassLoader for commons-text-api//org.apache.commons.text.StringSubstitutor.substitute(StringSubstitutor.java:1396)
	at PluginClassLoader for commons-text-api//org.apache.commons.text.StringSubstitutor.replaceIn(StringSubstitutor.java:1107)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.SecretSourceResolver.resolve(SecretSourceResolver.java:110)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.PrimitiveConfigurator.configure(PrimitiveConfigurator.java:48)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.tryConstructor(DataBoundConfigurator.java:164)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.instance(DataBoundConfigurator.java:75)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:274)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.configure(DataBoundConfigurator.java:81)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$doConfigure$16668e2$1(HeteroDescribableConfigurator.java:311)
	at PluginClassLoader for configuration-as-code//io.vavr.CheckedFunction0.lambda$unchecked$52349c75$1(CheckedFunction0.java:247)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.doConfigure(HeteroDescribableConfigurator.java:311)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$configure$2(HeteroDescribableConfigurator.java:88)
	at PluginClassLoader for configuration-as-code//io.vavr.control.Option.map(Option.java:391)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$configure$3(HeteroDescribableConfigurator.java:88)
	at PluginClassLoader for configuration-as-code//io.vavr.Tuple2.apply(Tuple2.java:240)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:86)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:57)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.tryConstructor(DataBoundConfigurator.java:164)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.instance(DataBoundConfigurator.java:75)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:274)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.configure(DataBoundConfigurator.java:81)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$doConfigure$16668e2$1(HeteroDescribableConfigurator.java:311)
	at PluginClassLoader for configuration-as-code//io.vavr.CheckedFunction0.lambda$unchecked$52349c75$1(CheckedFunction0.java:247)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.doConfigure(HeteroDescribableConfigurator.java:311)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$configure$2(HeteroDescribableConfigurator.java:88)
	at PluginClassLoader for configuration-as-code//io.vavr.control.Option.map(Option.java:391)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$configure$3(HeteroDescribableConfigurator.java:88)
	at PluginClassLoader for configuration-as-code//io.vavr.Tuple2.apply(Tuple2.java:240)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:86)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:57)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.tryConstructor(DataBoundConfigurator.java:156)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.instance(DataBoundConfigurator.java:75)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:274)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.impl.configurators.DataBoundConfigurator.check(DataBoundConfigurator.java:99)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:355)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.BaseConfigurator.check(BaseConfigurator.java:293)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:360)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.BaseConfigurator.check(BaseConfigurator.java:293)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.ConfigurationAsCode.lambda$checkWith$9(ConfigurationAsCode.java:868)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:811)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.ConfigurationAsCode.checkWith(ConfigurationAsCode.java:868)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:854)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:733)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:356)
	at PluginClassLoader for configuration-as-code//io.jenkins.plugins.casc.ConfigurationAsCode.init(ConfigurationAsCode.java:345)
Caused: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:109)
Caused: java.lang.Error
	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:115)
	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:304)
	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1149)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused: org.jvnet.hudson.reactor.ReactorException
	at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:290)
	at jenkins.InitReactorRunner.run(InitReactorRunner.java:49)
	at jenkins.model.Jenkins.executeReactor(Jenkins.java:1184)
	at jenkins.model.Jenkins.<init>(Jenkins.java:983)
	at hudson.model.Hudson.<init>(Hudson.java:102)
	at hudson.model.Hudson.<init>(Hudson.java:87)
	at hudson.WebAppMain$3.run(WebAppMain.java:249)
Caused: hudson.util.HudsonFailedToLoad
	at hudson.WebAppMain$3.run(WebAppMain.java:274)

24 is the hexadecimal ASCII code for the $ character.
From the stack trace, it appears to me that JCasC is running the decodeBase64() functionality, but the inner variable (which I had expected to be populated from the Kubernetes secret) is not getting resolved.

Full details below.

Version of Helm and Kubernetes

Helm:
  version.BuildInfo{Version:"v3.16.4", GitCommit:"7877b45b63f95635153b29a42c0c2f4273ec45ca", GitTreeState:"dirty", GoVersion:"go1.23.4"}`

Kubernetes:
  Client Version: v1.29.12
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.30.11-eks-bcf3d70

Chart version

jenkins-5.8.43 (latest at the time of issue raise, per https://artifacthub.io/packages/helm/jenkinsci/jenkins)

What happened?

Jenkins failed to start up due to an error in base64 decoding.

What you expected to happen?

Jenkins should have started up successfully.

How to reproduce it

(1.) AWS secret to Kubernetes Secret

I have an AWS secret containing various secrets I need to be mapped into Jenkins credentials, e.g. passwords, SSH keys, etc.

  • AWS secrets are stored as JSON blobs, so a secret value must be a single-line string to comply with JSON formatting rules.
  • The format of an SSH key (as consumed by some Jenkins credential providers) is a multiline string. To store a multiline string as single-line, we base64-encode it.
  • We therefore have a mixture of plaintext and base64-encoded credentials.
  • In order to store these in Kubernetes secrets, we use the kubernetes_secret Terraform provider, per https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret, putting everything in data.
  • The result is a Kubernetes secret that looks something like this:
apiVersion: v1
data:
  ec2-worker-ssh-key-base64: TFMwdExTMUNSVWRKVGlCUFVFVk9VMU5JSUZCU1NWWkJWRVVnUzBWWkxTMHRMUzBLTURGa1pXRmtZbVZsWm1SbFlXUmlaV1ZtWkdWaFpHSmxaV1prWldGa1ltVmxabVJsWVdSaVpXVm1aR1ZoWkdKbFpXWmtaV0ZrWW1WbFptUmxZV1JpWldWbVpHVmhaQW93TW1SbFlXUmlaV1ZtWkdWaFpHSmxaV1prWldGa1ltVmxabVJsWVdSaVpXVm1aR1ZoWkdKbFpXWmtaV0ZrWW1WbFptUmxZV1JpWldWbVpHVmhaR0psWldaa1pXRmtDakF6WkdWaFpHSmxaV1prWldGa1ltVmxabVJsWVdSaVpXVm1aR1ZoWkdKbFpXWmtaV0ZrWW1WbFptUmxZV1JpWldWbVpHVmhaR0psWldaa1pXRmtZbVZsWm1SbFlXUUtNRFJrWldGa1ltVmxabVJsWVdSaVpXVm1aR1ZoWkdKbFpXWmtaV0ZrWW1WbFptUmxZV1JpWldWbVpHVmhaR0psWldaa1pXRmtZbVZsWm1SbFlXUmlaV1ZtWkdWaFpBb3dOV1JsWVdSaVpXVm1aR1ZoWkdKbFpXWmtaV0ZrWW1WbFptUmxZV1JpWldWbVpHVmhaR0psWldaa1pXRmtZbVZsWm1SbFlXUmlaV1ZtWkdWaFpEQXdDaTB0TFMwdFJVNUVJRTlRUlU1VFUwZ2dVRkpKVmtGVVJTQkxSVmt0TFMwdExRbz0K
  my-password: aW5zZWN1cmVQYXNzd29yZAo=
kind: Secret

A key point here is that ec2-worker-ssh-key-base64 is double-base64-encoded above.

(2.) Kubernetes Secret to Helm chart variables

Per the documentation, I have configuration like this:

controller:
  additionalExistingSecrets:
    - name: jenkins-additional-secrets
      keyName: my-password
    - name: jenkins-additional-secrets
      keyName: ec2-worker-ssh-key-base64

This generates Helm variables named jenkins-additional-secrets-my-password and jenkins-additional-secrets-ec2-worker-ssh-key-base64.

(3.) Helm chart variables to JCasC Credential configuration

controller:
  JCasC:
    configScripts:
      jenkins-casc-configs: |
        credentials:
          system:
            domainCredentials:
              - credentials:
                  - basicSSHUserPrivateKey:
                      id: "ec2-ssh-key-credential"
                      description: "jenkins Ed25519 SSH key, with username set to jenkins"
                      username: "jenkins"
                      usernameSecret: false
                      privateKeySource:
                        directEntry:
                          privateKey: ${decodeBase64:${jenkins-additional-secrets-ec2-worker-ssh-key-base64}}
                      scope: GLOBAL

Then try deploying this, and BOOM.

Without the ${decodeBase64:...} wrapper, secret values are interpolated just fine (see the workaround described below).

Anything else we need to know?

There is a workaround for this issue, that I am currently using: if you change the configuration of the kubernetes_secret Terraform provider to populate binary_data with any secret values that are already base64-encoded, this skips the secondary base64-encoding that happens to values present in data. Therefore, when Jenkins base64-decodes these, they are already in the plain form.

However:

  1. this is additional complexity to maintain
  2. it seems that I should be able to avoid this by using the decodeBase64() functionality described in https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/docs/features/secrets.adoc#additional-variable-substitution
  3. other Jenkins credential types require a base64-encoded input value (e.g. file, for secretBytes); it would be nice to be able to clearly handle decoding (or encoding!) where needed in the credential management part of the JCasC section of my Helm values files, instead of being forced to handle it upstream
  4. perhaps a special incantation is needed for this to work? (in which case, others may benefit from reading the replies to the issue, and perhaps from improved documentation!)
  5. perhaps this "just isn't possible" for some reason? (in which case, I'd like to understand why!)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions