Add Maven Profile To Config Server To Include spring-vault-core

[arjavdongaonkar]

Description:

I am currently using Spring Cloud Kubernetes Config Server (v3.2.0), but I am facing an issue with HashiCorp Vault token renewal. When running the Spring Cloud Config Server normally (v4.2.0), the Vault token renewal activity is automatically triggered. However, with the Spring Cloud Kubernetes Config Server (v3.2.0), the token renewal is not triggered, resulting in the missing X-Config-Token header in requests.

Expected Behavior:

The Vault token renewal should be automatically triggered when using Spring Cloud Kubernetes Config Server, and the X-Config-Token header should be included in the requests.

Environment:

• Spring Cloud Kubernetes Config Server version: 3.2.0
• Spring Cloud Config Server version (when working normally): 4.2.0

Investigation:

After further investigation, I discovered that the spring-vault-core dependency must be included in the application's classpath to trigger LifecycleAwareSessionManager (which is responsible for the token renewal process).

On Spring Cloud Config Server (v4.2.0), I see the following log line:
o.s.v.a.LifecycleAwareSessionManager - Scheduling Token renewal

However, this log line is not seen when using Spring Cloud Kubernetes Config Server (v3.2.0), indicating that the token renewal is not being triggered.

logs:

` \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::                (v3.4.0)

07:16:29.522 [main] INFO  o.s.c.k.c.KubernetesConfigServerApplication - Starting KubernetesConfigServerApplication v3.2.0 using Java 21.0.5 with PID 1 (/usr/src/app/spring-cloud-kubernetes-configserver-3.2.0.jar started by root in /usr/src/app) 
07:16:29.526 [main] INFO  o.s.c.k.c.KubernetesConfigServerApplication - The following 2 profiles are active: "kubernetes", "composite" 
07:16:37.401 [main] INFO  o.s.c.context.scope.GenericScope - BeanFactory id=18b381b0-f077-3064-b6f3-ccc47cc63c6b 
07:16:40.020 [main] INFO  o.s.b.w.e.tomcat.TomcatWebServer - Tomcat initialized with port 8888 (http) 
07:16:40.112 [main] INFO  o.a.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8888"] 
07:16:40.118 [main] INFO  o.a.catalina.core.StandardService - Starting service [Tomcat] 
07:16:40.119 [main] INFO  o.a.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/10.1.33] 
07:16:40.424 [main] INFO  o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext 
07:16:40.425 [main] INFO  o.s.b.w.s.c.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 10693 ms 
07:16:43.305 [main] INFO  o.h.validator.internal.util.Version - HV000001: Hibernate Validator 8.0.1.Final 
07:16:51.826 [main] INFO  o.s.b.a.e.web.EndpointLinksResolver - Exposing 16 endpoints beneath base path '/actuator' 
07:16:52.821 [main] INFO  o.a.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8888"] 
07:16:52.911 [main] INFO  o.s.b.w.e.tomcat.TomcatWebServer - Tomcat started on port 8888 (http) with context path '/' 
07:16:53.041 [main] INFO  o.s.c.k.c.KubernetesConfigServerApplication - Started KubernetesConfigServerApplication in 35.837 seconds (process running for 39.806) 
07:16:55.831 [http-nio-8888-exec-1] INFO  o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring DispatcherServlet 'dispatcherServlet' 
07:16:55.833 [http-nio-8888-exec-1] INFO  o.s.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet' 
07:16:55.835 [http-nio-8888-exec-1] INFO  o.s.web.servlet.DispatcherServlet - Completed initialization in 2 ms
07:17:35.174 [http-nio-8888-exec-9] WARN  o.s.c.c.s.e.EnvironmentController - Error getting the Environment with name=communication-service profiles=postgres,rabbitmq,redis,domains,access-keys,s3,kafka,communication-service,secrets label=null includeOrigin=false 
java.lang.IllegalArgumentException: Missing required header in HttpServletRequest: X-Config-Token
`

Can someone help with this issue and provide guidance on how to enable Vault token renewal for Spring Cloud Kubernetes Config Server (v3.2.0)?

[ryanjbaxter]

Are using the docker image for the config server from DockerHub?

Can you provide your deployment yaml and any other configuration you are setting for the config server?

[arjavdongaonkar]

Hey @ryanjbaxter,

Sure, here’s the information you requested:

Docker Image:
I am building the image from the source code. Here’s the relevant snippet from the Dockerfile:

RUN git clone --branch v3.2.0 --depth 1 https://github.com/spring-cloud/spring-cloud-kubernetes.git .
WORKDIR $APP_HOME/spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configserver
RUN mvn package org.springframework.boot:spring-boot-maven-plugin:repackage -DskipTests -Dskip.build.image=true

The resulting JAR file is then executed with Java.

Deployment YAML:
Below is the Deployment YAML I’m using. It includes two active profiles: kubernetes and composite (for Vault and Git configurations).

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        image: <img_tag>
        env:
        - env:
        - name: SPRING_PROFILES_ACTIVE
          value: "composite"
        - name: SPRING_CLOUD_CONFIG_SERVER_VAULT_KUBERNETES_ROLE
          value: <vault-config-server-role>
        - name: SPRING_CLOUD_CONFIG_SERVER_VAULT_HOST
          value: <Vault_host>
        - name: SPRING_CLOUD_CONFIG_SERVER_VAULT_AUTHENTICATION
          value: "KUBERNETES"
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_TYPE
          value: "vault"
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_BACKEND
          value: "secret"
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_KVVERSION
          value: '2'
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_HOST
          value: <Vault_host>
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_TYPE
          value: "git"
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_DEFAULTLABEL
          value: ${GIT_BRANCH}
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_URI
          value: ${GIT_URI}
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_SEARCHPATHS
          value: ${GIT_SEARCHPATHS}
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_PASSWORD
          value: ${GIT_TOKEN}
        - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_USERNAME
          value: ${GIT_USERNAME}
        - name: SPRING_PROFILES_INCLUDE
          value: 'kubernetes'
        - name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLEAPI
          value: 'true'
        - name: SPRING_CLOUD_KUBERNETES_CONFIG_ENABLEAPI
          value: 'false'
        image: <Image>

it works when we provide this as well

        - name: SPRING_CLOUD_CONFIG_SERVER_VAULT_TOKEN
          value: <Vault_token>

We have attached required service account and roles to this deployment (even joined this in role binding).
Followed this doc: https://docs.spring.io/spring-cloud-kubernetes/reference/spring-cloud-kubernetes-configserver.html
Let me know if you need further clarification or additional details!

[arjavdongaonkar]

Hi @ryanjbaxter,

I had a couple of questions that I’d love your input on:

1. Accessing Secrets in Config Server API
   When calling the Config Server to fetch configurations, we currently need to explicitly include secrets in the profile, like this:
   curl --location 'http://localhost:8888/ab-service/ab-service,secrets'
   Since Kubernetes Secrets are similar to Vault in nature, is there a way to configure the Config Server to fetch them by default without explicitly including them in the profiles (similar to how Vault configurations are handled)?
   Is it possible to treat Kubernetes Secrets as a composite type? This way, we could configure them alongside other sources, such as Vault or Git, allowing more flexibility in configuration management.

2. Secrets Priority in propertySources
   In the current setup, Kubernetes Secrets appear at the bottom of the propertySources list in the API response.
   For example:

"propertySources": [
  { "name": "vault:application" },
  { "name": "secret:ab-service-secrets" }
]

Can we configure the order of propertySources so that Kubernetes Secrets appear at the top of the list? This would ensure that values from Secrets take precedence, allowing other configurations to resolve their values using Secrets.

[ryanjbaxter]

I am going to focus on one issue at a time we can come back to your other questions after we have a good idea about your original issue.

You said

it works when we provide this as well
- name: SPRING_CLOUD_CONFIG_SERVER_VAULT_TOKEN
value: <Vault_token>

So does that mean if you set that environment variable everything works on Kubernetes?

[arjavdongaonkar]

Yes, But that's the static token, and i dont want to provide that in env vars.

[ryanjbaxter]

A couple more questions.

1. Do you have spring-vault-core on the classpath of your vanilla config server where vault token configuration works?

2. When deploying your kubernetes config server to kubernetes you have

- name: SPRING_CLOUD_CONFIG_SERVER_VAULT_AUTHENTICATION
          value: "KUBERNETES"

This would set the authentication method to kubernetes not token. That is a little confusing to me.

[arjavdongaonkar]

1. Classpath Dependency:
   In the original Spring Cloud Config Server, spring-vault-core is present on the classpath. However, in the Spring Cloud Kubernetes Config Server, it is missing. When I manually added spring-vault-core to the pom.xml before building, the issue was resolved. However, I want to avoid manually modifying dependencies.

2. Authentication Method:
   The Spring Cloud Config Server (composite type: Vault, Git) works fine with SPRING_CLOUD_CONFIG_SERVER_VAULT_AUTHENTICATION=KUBERNETES, so I retained that configuration when deploying spring-cloud-kubernetes-config-server.
   When I change it to TOKEN, the original config server fails with:

Caused by: java.lang.IllegalArgumentException: The 'spring.cloud.config.server.vault.token' property must be provided when the TOKEN authentication method is specified.

[ryanjbaxter]

1. Classpath Dependency:
   In the original Spring Cloud Config Server, spring-vault-core is present on the classpath. However, in the Spring Cloud Kubernetes Config Server, it is missing. When I manually added spring-vault-core to the pom.xml before building, the issue was resolved. However, I want to avoid manually modifying dependencies.

Are you adding it? Because it is optional by default.
https://github.com/spring-cloud/spring-cloud-config/blob/main/spring-cloud-config-server/pom.xml#L61-L65

2. Authentication Method:
   The Spring Cloud Config Server (composite type: Vault, Git) works fine with SPRING_CLOUD_CONFIG_SERVER_VAULT_AUTHENTICATION=KUBERNETES, so I retained that configuration when deploying spring-cloud-kubernetes-config-server.
   When I change it to TOKEN, the original config server fails with:

Caused by: java.lang.IllegalArgumentException: The 'spring.cloud.config.server.vault.token' property must be provided when the TOKEN authentication method is specified.

Why are you setting it to be KUBERNETES if you want it to use TOKEN?

[arjavdongaonkar]

When building the spring-cloud-config-server as a vanilla setup in my Dockerfile, I am using the default POM.xml from the Spring Cloud Config Server repository. Although spring-vault-core is an optional dependency, it still gets included in the classpath by default.

However, in the case of spring-cloud-kubernetes-config-server, the optional dependency does not get added to the classpath, causing token refresh failures.

As a workaround (kind of hacky), I manually added the spring-vault-core dependency to the Kubernetes Config Server POM.xml, which resolved the problem and allowed automatic token refresh:

RUN git clone --branch v3.2.0 --depth 1 https://github.com/spring-cloud/spring-cloud-kubernetes.git .

WORKDIR $APP_HOME/spring-cloud-kubernetes-controllers/spring-cloud-kubernetes-configserver

# Add the spring-vault-core dependency below the spring-cloud-config-server dependency
RUN apk add --no-cache bash && \
    sed -i '/<artifactId>spring-cloud-config-server<\/artifactId>/a \
        <dependency>\n\
            <groupId>org.springframework.vault</groupId>\n\
            <artifactId>spring-vault-core</artifactId>\n\
        </dependency>' pom.xml

RUN mvn package org.springframework.boot:spring-boot-maven-plugin:repackage -DskipTests -Dskip.build.image=true

While this approach works, i guess manually modifying the POM is not good.
I’d love to know if there’s a more elegant way to achieve this, since using Vault with Kubernetes secrets is very well a use case.
Thank you!!

@ryanjbaxter, could you please provide your insights on this as well? Your guidance would be greatly helpful.

1. Accessing Secrets in Config Server API
   When calling the Config Server to fetch configurations, we currently need to explicitly include secrets in the profile, like this:
   curl --location 'http://localhost:8888/ab-service/ab-service,secrets'
   Since Kubernetes Secrets are similar to Vault in nature, is there a way to configure the Config Server to fetch them by default without explicitly including them in the profiles (similar to how Vault configurations are handled)?
   Is it possible to treat Kubernetes Secrets as a composite type? This way, we could configure them alongside other sources, such as Vault or Git, allowing more flexibility in configuration management.
2. Secrets Priority in propertySources
   In the current setup, Kubernetes Secrets appear at the bottom of the propertySources list in the API response.
   For example:

"propertySources": [
  { "name": "vault:application" },
  { "name": "secret:ab-service-secrets" }
]

Can we configure the order of propertySources so that Kubernetes Secrets appear at the top of the list? This would ensure that values from Secrets take precedence, allowing other configurations to resolve their values using Secrets.

[arjavdongaonkar]

I’d love to know if there’s a more elegant way to achieve this, since using Vault with Kubernetes secrets is very well a use case.

Please let me know if its a correct approach and can be added.
#1864

[ryanjbaxter]

When building the spring-cloud-config-server as a vanilla setup in my Dockerfile, I am using the default POM.xml from the Spring Cloud Config Server repository. Although spring-vault-core is an optional dependency, it still gets included in the classpath by default.

Can you please provide me with a sample of your project than either as a git repo or zip file so I can look at it?

I’d love to know if there’s a more elegant way to achieve this, since using Vault with Kubernetes secrets is very well a use case.

Not at the moment, we are not going to add spring-vault-core as a dependency when it is very specific to the vault use case. In addition you can do TOKEN authentication without spring-vault-core, by supplying the token as a header in the request to the config server, see https://docs.spring.io/spring-cloud-config/reference/server/environment-repository/vault-backend.html. This is why you are seeing the error when deploying to k8s. The only reason it works in your vanilla config server is because somehow spring-core-vault is on the classpath (which it is not by default so it is somehow being added).

Please open 2 separate issues for the other questions/requests you had.

[arjavdongaonkar]

Thank you for the clarification.

My vanilla Config Server setup is simply a Dockerfile where I build the Config Server using mvn package and then run the JAR. You can find the Dockerfile here

Please let me know if its a correct approach and can be added.
#1864

I have created a PR by forking the Spring Cloud Kubernetes Config Server, to introduce a useVault profile to conditionally add the spring-vault-core dependency. Please share your thoughts on this approach.

For my other questions, I'll open separate issues as suggested.

[ryanjbaxter]

OK that makes sense. Typically when people create a config server they ass the dependency

<dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-config-server</artifactId>
    </dependency>

And then add @EnableConfigServer to their main application class. In that case spring-vault-core is not on the classpath.

I have added comments to #1864.

[arjavdongaonkar]

Great! Thanks
I have created a PR against 3.1.x brach, Please find it here