Vault Agent auto-auth re-authentication appends headers instead of replacing existing header keys

[debben]

Vault Agent auto-auth appears to accumulate headers on the API client with each iteration. This breaks auto-auth methods like Kerberos.

Environment:
Vault v1.19.13+ent (27ea1d934ebaa51cab4d1e8c9d8b90e63dfae0dc), built 2026-01-06T01:07:56Z

• Vault Version: v1.19.13+ent (27ea1d934ebaa51cab4d1e8c9d8b90e63dfae0dc), built 2026-01-06T01:07:56Z
• Operating System/Architecture: Windows (Vault Agent), Linux (Vault Server)

Vault Config File:

pid_file = "C:\\vault-agent\\agent.pid"  # Specify the path to the PID file for Vault Agent

auto_auth {
  method "kerberos" {
    mount_path = "auth/kerberos"

    config = {
      krb5conf_path = "C:\\path\\to\\krb5.conf"
      keytab_path   = "C:\\path\\to\\<REDACTED>.keytab"
      username      = "<REDACTED>"
      service       = "<REDACTED>"
      realm         = "<REDACTED>"
      disable_fast_negotiation = true
    }
  }
  sink "file" {
    config = {
      path = "vault_sink_token"
    }
  }
}


exit_after_auth = false

log_level = "debug"

vault {
  address = "http://localhost:8200"
}

Startup Log Output:

Timestamps are in Eastern Time:

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.19.13+ent, built 2026-01-06T01:07:56Z
             Version Sha: 27ea1d934ebaa51cab4d1e8c9d8b90e63dfae0dc

2026-07-01T11:55:33.625-0400 [INFO]  agent.sink.file: creating file sink
2026-07-01T11:55:33.628-0400 [INFO]  agent.sink.file: file sink configured: path=vault_sink_token mode=-rw-r----- owner=-1 group=-1
2026-07-01T11:55:33.631-0400 [DEBUG] agent: would have sent systemd notification (systemd not present): notification=READY=1
2026-07-01T11:55:33.631-0400 [INFO]  agent.exec.server: starting exec server
2026-07-01T11:55:33.631-0400 [INFO]  agent.exec.server: no env templates or exec config, exiting
2026-07-01T11:55:33.631-0400 [INFO]  agent.auth.handler: starting auth handler
2026-07-01T11:55:33.631-0400 [INFO]  agent.auth.handler: authenticating
2026-07-01T11:55:33.631-0400 [INFO]  agent.sink.server: starting sink server
2026-07-01T11:55:33.631-0400 [INFO]  agent.template.server: starting template server
2026-07-01T11:55:33.632-0400 [INFO]  agent.template.server: no templates found
2026-07-01T11:55:35.113-0400 [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2026-07-01T11:55:35.113-0400 [INFO]  agent.auth.handler: starting renewal process
2026-07-01T11:55:35.116-0400 [INFO]  agent.sink.file: token written: path=vault_sink_token
2026-07-01T12:02:44.482-0400 [INFO]  agent.auth.handler: lifetime watcher done channel triggered, re-authenticating
2026-07-01T12:02:45.482-0400 [INFO]  agent.auth.handler: authenticating
2026-07-01T12:02:48.396-0400 [ERROR] agent.auth.handler: error authenticating:
  error=
  | Error making API request.
  | 
  | URL: PUT http://localhost:8200/v1/auth/kerberos/login
  | Code: 401. Errors:
  | 
  backoff=830ms
2026-07-01T12:02:49.232-0400 [INFO]  agent.auth.handler: authenticating
2026-07-01T12:02:49.916-0400 [ERROR] agent.auth.handler: error authenticating:
  error=
  | Error making API request.
  | 
  | URL: PUT http://localhost:8200/v1/auth/kerberos/login
  | Code: 401. Errors:
  | 
  backoff=830ms
2026-07-01T12:02:51.561-0400 [INFO]  agent.auth.handler: authenticating

Expected Behavior:

When the auto-auth method re-authenticates, it should succeed. A new call to Authenticate(...) should return a fresh header, and only that header should be sent.

Actual Behavior:

Subsequent auto-auth attempts fail, and the Vault server reports:

vault-server  | 2026-07-01T16:02:48.389Z [INFO]  auth.kerberos.auth_kerberos_<redacted>: <redacted> - SPNEGO validation error: defective token detected: KRB Error: (37) KRB_AP_ERR_SKEW Clock skew too great - clock skew with client too large. greater than 5m0s seconds
vault-server  | 2026-07-01T16:02:49.951Z [INFO]  auth.kerberos.auth_kerberos_<redacted>: <redacted> - SPNEGO validation error: defective token detected: KRB Error: (37) KRB_AP_ERR_SKEW Clock skew too great - clock skew with client too large. greater than 5m0s seconds

Using an HTTP traffic inspector to inspect traffic between a local Vault Agent instance and a local Vault server container, the initial successful request to /v1/auth/kerberos/login includes the expected Authorization: Negotiate ... header. Each subsequent failed call to the login endpoint from Vault Agent has multiple Authorization: Negotiate ... headers on the request, appearing to accumulate with each retry. The KRB_AP_ERR_SKEW appears to occur because Vault server reads the first Authorization: Negotiate ... value, which is a replayed header from an earlier request.

Steps to Reproduce:

• Configure a local Vault server with Kerberos auth (for easier reproduction, use a low default_lease_ttl, for example 10m).
• Configure Vault Agent auto-auth to use Kerberos and point it at that Vault server instance.

Important Factoids:

I encountered this with the Kerberos auto-auth method, but I believe this applies to any auth method implementing Authenticate that returns headers.

References:

auth.go L362-L366 where headers are added (but not overwritten if a header already exists)

[dosubot]

Hi @debben! I'm Dosu and I'm helping the HashiCorp, an IBM Company team.

You've correctly identified the bug. In auth.go at lines 362-366, headers returned by the auth method are appended using AddHeader() (which calls Go's http.Header.Add()) rather than replacing existing keys [1]. This causes the stale Authorization: Negotiate ... header from the initial authentication to persist alongside newly generated ones on each re-auth cycle.

There is an existing open PR that fixes this: #31723. The fix replaces the AddHeader() loop with a single clientToUse.SetHeaders(header) call, ensuring headers are overwritten rather than accumulated on subsequent re-authentication attempts.

Since you're running an Enterprise version (v1.19.13+ent), I'd recommend opening a support ticket at https://support.hashicorp.com/hc/en-us/requests/new to track this for your environment and inquire about when the fix might land in an enterprise release.

Written by @dosu, approved by heatherezell

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.