Vault Agent: auto-auth self-healing not wired for proxy/cache requests (missing channel assignment in `command/agent.go`)

[khailequang334]

Hi team,

I'd like to report what appears to be a missing channel assignment in command/agent.go that prevents the auto-auth self-healing feature from working when Vault Agent is used with the caching/proxy listener. I wanted to check with you whether this is intentional (given Agent's API proxy is deprecated in favor of Vault Proxy) or if it was an oversight.

Summary

The auto-auth self-healing feature (introduced in v1.17.0 via #26307) correctly wires the invalidTokenErrCh and authInProgress channels in command/proxy.go, but the equivalent assignment is missing in command/agent.go. This means the self-healing works for vault proxy but not for vault agent when using the cache/listener.

Code References (v1.21.4)

In command/proxy.go, the local variables are reassigned to the auth handler's channels after ah is created:

https://github.com/hashicorp/vault/blob/v1.21.4/command/proxy.go#L578-L579

authInProgress = ah.AuthInProgress
invalidTokenErrCh = ah.InvalidToken

In command/agent.go, the same variables are created at:

https://github.com/hashicorp/vault/blob/v1.21.4/command/agent.go#L552-L553

authInProgress := &atomic.Bool{}
invalidTokenErrCh := make(chan error)

But they are never reassigned to ah.AuthInProgress / ah.InvalidToken. They are passed directly to ProxyHandler as orphaned local variables:

https://github.com/hashicorp/vault/blob/v1.21.4/command/agent.go#L685-L687

For reference, the auth handler creates its own buffered channel and atomic bool that are meant to be used:

https://github.com/hashicorp/vault/blob/v1.21.4/command/agentproxyshared/auth/auth.go#L99-L101

InvalidToken:    make(chan error, 1),  // buffered
AuthInProgress:  &atomic.Bool{},

And the auth handler listens on ah.InvalidToken to trigger re-auth:

https://github.com/hashicorp/vault/blob/v1.21.4/command/agentproxyshared/auth/auth.go#L607

case <-ah.InvalidToken:
    ah.logger.Info("invalid token found, re-authenticating")
    break LifetimeWatcherLoop

Observed Behavior

When running vault agent (v1.17.3 tested, also verified in v1.21.4 source) with auto_auth + cache + listener:

1. Revoke the agent's token on the server side
2. Send a request through the agent's cache listener
3. Agent logs: "proxy received an invalid token error" (from handler.go#L77)
4. But the agent never logs "invalid token found, re-authenticating" (from auth.go#L608)
5. The client request hangs until timeout — no HTTP response is returned

The hang occurs because invalidTokenErrCh is an unbuffered channel with no reader. The send at handler.go#L81 blocks the HTTP handler goroutine, preventing the response from being flushed.

Expected Behavior

Same as vault proxy — after detecting an invalid token, the agent should trigger re-authentication and retry with a fresh token.

Additional Impact

• Each request that hits this path creates a blocked goroutine (~4-8 KB) that is never cleaned up, leading to a small but cumulative resource leak under sustained retry load
• The authInProgress variable is also not wired (agent.go L552), so the handler always reads false and always attempts the channel send, even during actual re-authentication
• Switching to vault proxy is not always feasible — the Vault Agent Injector for Kubernetes only supports injecting vault agent, not vault proxy (see vault-k8s #495)

Affected Versions

v1.17.0 through v1.21.4, and current main branch (v2.0.2).

Possible Fix

Add the two missing assignments in command/agent.go, after ah is created (around line 600), matching what command/proxy.go already does:

authInProgress = ah.AuthInProgress
invalidTokenErrCh = ah.InvalidToken

Thank you for your time, and for the great work on the self-healing feature.

[dosubot]

Hi @khailequang334! I'm Dosu and I'm helping the HashiCorp, an IBM Company team.

Your analysis is correct. I can confirm that in command/agent.go, authInProgress and invalidTokenErrCh are initialized as local variables at lines 553–554 [1] but are never reassigned to ah.AuthInProgress / ah.InvalidToken after the auth handler is created at lines 586–600 [2]. They are passed directly to the proxy handler as orphaned locals at lines 706–708 [3].

In contrast, command/proxy.go correctly reassigns these at lines 578–579:

authInProgress = ah.AuthInProgress
invalidTokenErrCh = ah.InvalidToken

This appears to have been an oversight when PR #26307 extended the self-healing feature to Vault Proxy — the fix was applied to proxy.go but not mirrored back to agent.go. I did not find any existing issues or PRs tracking this bug.

Your proposed fix (adding the two reassignment lines in agent.go after ah is created) matches exactly what proxy.go does and looks like the correct resolution. The additional observations about the goroutine leak from the unbuffered channel send and the authInProgress always reading false are also well-documented consequences of this missing wiring.

Tagging @heatherezell to flag this for product and engineering review.

Written by @dosu, approved by heatherezell

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[VishalGawade1]

Hi, I'd like to pick this up if it's open for contribution.
Also, @khailequang334 raised whether this is intended behavior, since Agent's API proxy is deprecated in favor of Vault Proxy. Do you want this fixed, or left as-is? Just checking the direction first

[heatherezell]

Hi @VishalGawade1! I think if you'd like to take on a PR, this is a good one to choose. Deprecated or no, I personally lean towards "let's fix the code and when it's time to be truly removed, we can clean it up then but until then the behavior should probably match across the functionality". Worst case, the PR may not be accepted, but I know I would appreciate efforts to help get this across the finish line for our users who are still using this. Thanks so much!

Please feel free to ping me directly with an "at" once the PR is up. Also, please feel free to leverage Dosu. It has access to the code base, documentation, and support KBs and can help guide you.

[VishalGawade1]

Hi, I would appreciate if you could take a look at my PR and let me know if you want me change anything!

[khailequang334]

Hi @VishalGawade1, thanks for the quick fix. I also had a PR prepared for this issue, but since you’ve already raised it, I think we can proceed with the review on your PR instead 👍
BTW, your PR reflects exactly my proposed fix—with well-tested changes. LGTM 💯

[khailequang334]

Hi @heatherezell, thanks for the review and bug confirmation.
Regarding the vault-agent API proxy deprecation, could you confirm whether the team has a real deprecation plan in place? Does this mean the entire vault-agent product is deprecated? If yes, when will it happen, and what’s the alternative after we deprecate it?

[heatherezell]

Hello @khailequang334, Vault Agent API has been deprecated and Vault Proxy has all the functionality of the Vault Agent API. I do not know exactly when the Vault Agent API functionality will be removed from Vault, but I do know we encourage all our users to use Vault Proxy instead.

[VishalGawade1]

Hi @khailequang334 , really sorry, I didn't know you had already raised a PR for this. Didn't mean to take up your issue. Let me know if you want me to make any changes

[khailequang334]

Hi @heatherezell , thanks for sharing the information. But cmiiw, after the Vault Agent API is fully removed, if user’s usecase includes both secret template and request proxy, would they need to run both vault-agent for secret template and vault-proxy for request proxy on the client side?

[khailequang334]

Hi @VishalGawade1, no worries, you did it well, the PR looks good to me. Thankss!

[VishalGawade1]

Hi, are we good to close the PR?