How to preemptively refresh credentials (sts, eks pod, java)?

[kncesarini]

I'm using the java sdk v2. My spring boot app is running in EKS with a pod identity, so credentials are supplied automagically by sts with the Default credentials provider chain. The app reads and writes things to dynamodb- this is how the client is created.

DynamoDbClient.builder().region(Region.of(region)).build();

The issue I'm having is that these credentials don't seem to be refreshed proactively. When the app starts up after a reboot or deploy, the first request will be slower, since the app finds it has no credentials, fetches credentials from STS, then does the dynamodb request. If the app runs long enough for the credentials to expire, I assume this happens again, but haven't tested that specific scenario thoroughly.

What I want is for this refresh to happen automatically before any dynamodb requests are made. When the app starts, credentials should be fetched before the app is "ready". When credentials are about to expire, the app should refresh them before they do. Is there a way to accomplish this? I would hate to e.g. have to use an iam user just to work around this performance annoyance.

[debora-ito]

EKS with Pod Identity is part of the ContainerCredentialsProvider, and it has the ability to refresh credentials when they are about to expire. For additional assistance and make the credential refreshing to execute in the background, you can enable the asyncCredentialUpdateEnabled option:

        ContainerCredentialsProvider containerProvider = ContainerCredentialsProvider.builder()
                .asyncCredentialUpdateEnabled(Boolean.TRUE)
                .build();

This option is disabled by default, which means if you use the default credential provider chain this will be disabled. But you can set ContainerCredentialsProvider to be used explicitly in the client.

[mgs255]

Hi @debora-ito

If we use the default credential provider but enable asyncCredentialUpdateEnabled should that also work too? We often see workloads using Pod Identity seemingly lose permissions to make calls to apis which previously were successful. We are also using spring-boot with an up to date version of the V2 SDK.

This is the sort of pattern we might use to create a S3 client:

    private getS3Client( final String region )
    {
        this.amazonS3 = S3Client.builder()
            .region( Region.of( region ) )
            .credentialsProvider( DefaultCredentialsProvider.builder().asyncCredentialUpdateEnabled( true ).build() )
            .build();
    }

However, we don't seem to see any noticeable difference in the failure rate after enabling async credential updates. For example we might see this sort of error in the logs:

software.amazon.awssdk.services.s3.model.S3Exception: User: arn:aws:sts::<accountId>:assumed-role/<clusterName>-worker-nodes/i-088c0dfb949b5df78 is not authorized to perform: s3:GetLifecycleConfiguration on resource: "arn:aws:s3:::<rolePrefix>-preview-dev" because no identity-based policy allows the s3:GetLifecycleConfiguration action (Service: S3, Status Code: 403, Request ID: TYBCZ65PCQSSDRMQ, Extended Request ID: lPk1Q5Go0+KsH231Brwmzdg0751NUjDksd1wIi+QdzZGm6hoMQk7hk+BxVIyVcRBZaGmJR9nYB4=)