VSO continues using cached tokens with stale policies after Vault role policies are updated, causing persistent 403 errors #1185
Description
Describe the bug
When a new policy is added to a vault role, and a vaultstaticsecret ( or other secret types ) are updated / created to pull in a secret from the new policy, the VSO client / token remains cached even though it gets a 403 error.
Today, when the client receives a 403 error, it taints the client. https://github.com/hashicorp/vault-secrets-operator/blob/main/controllers/vaultstaticsecret_controller.go#L164-L165
In this specific use-case, the token is still valid, but the cached token does not have the additional policy. So when it goes to check if the tainted client is valid or not, it returns valid and remains cached. https://github.com/hashicorp/vault-secrets-operator/blob/main/vault/client.go#L290-L294
This would then cause the vaultstaticsecret to not reconcile and the client to return 403s errors until the token TTL expires.
Our standard operation today is to go delete the vaultauth object and recreate it to get a new token instead of waiting for the TTL to expire.
To Reproduce
Steps to reproduce the behavior:
- Deploy standard vso setup with kubernetes auth and role
- Create a static secret that leverages that role
- Add an additional policy to the role and update the vaultstaticsecret to now pull in a secret from a path on the additional poliy
- You can view the logs or describe the vaultstaticsecret to see the 403 error
- It will continue to error until the token has expired and vso have to retrieve a new one
- Alternatively, you can delete the vault auth object and recreate it to have it reconcile properly
Expected behavior
I would expect that when the client is tainted for a 403 that it is evicted from cache and retrieves a new token.
Environment
- vault-secrets-operator version: 1.0.1
Additional context
I'm happy to contribute a PR if this functionality makes sense