Restricting CRT s3 client memory usage?

[Lunkers]

Hi! I have a CRT s3 client and transfer manager setup like this:

s3Client = S3AsyncClient.crtBuilder()
        .credentialsProvider(
            StaticCredentialsProvider.create(
                AwsBasicCredentials.create(
                    s3Endpoint.accessKey,
                    s3Endpoint.secretKey
                )
            )
        )
        .maxConcurrency(s3Properties.maxConcurrent)
        .region(Region.US_EAST_1)
        .endpointOverride(s3Endpoint.url)
        .forcePathStyle(true)
        .checksumValidationEnabled(false)
        .build()
        
 transferManager = S3TransferManager.builder()
    .s3Client(s3Client)
    .build()

When running the code in Kubernetes and trying to upload a large file (80-100GB), the pod uses way more memory than it's been assigned to do (>9GB in a pod with 8GB limit), and subsequently gets OOM-killed; since the java heap is smaller than 1GB, I'm suspecting that the CRT libraries is the reason for the high memory consumption. The same behavior occurs when uploading the same file using the CLI.

Is there any way to restrict how much memory the CRT client is allowed to use? The closes I've seen is setting maxConcurrency, but it seems very roundabout.

[Lunkers]

After some thinking, this seems very much like unintended behavior. I've created an issue for it, see #4034

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.