`@aws-sdk/client-s3` may produce malformed S3 streaming (`aws-chunked`) data

[mxxk]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

When S3 receives a streaming body (e.g., STREAMING-AWS4-HMAC-SHA256-PAYLOAD), it requires a minimum chunk size of 8 KiB (see: Signature Calculations for the Authorization Header: Transferring Payload in Multiple Chunks (Chunked Upload) (AWS Signature Version 4)).

Consolidating chunks smaller than 8 KiB is performed by @smithy/util-stream:createBufferedReadable, but createBufferedReadable is only used if requestStreamBufferSize is at least 8 KiB:

aws-sdk-js-v3/packages/middleware-flexible-checksums/src/flexibleChecksumsMiddleware.ts

Lines 128 to 139 in 3b6a4d9

	updatedBody = getAwsChunkedEncodingStream(
	typeof config.requestStreamBufferSize === "number" && config.requestStreamBufferSize >= 8 * 1024
	? createBufferedReadable(requestBody, config.requestStreamBufferSize, context.logger)
	: requestBody,
	{
	base64Encoder,
	bodyLengthChecker,
	checksumLocationName,
	checksumAlgorithmFn,
	streamHasher,
	}
	);

The issue is that the default S3Client configuration does not initialize requestStreamBufferSize, so it defaults to 0 and streaming input data is not buffered. This causes multiple problems:

• If the input stream yields chunks smaller than 8 KiB, S3 rejects the request:

  <?xml version="1.0" encoding="UTF-8"?>
  <Error>
    <Code>InvalidChunkSizeError</Code>
    <Message>Only the last chunk is allowed to have a size less than 8192 bytes</Message>
    <Chunk>2</Chunk>
    <BadChunkSize>1</BadChunkSize>
    <RequestId>MFV9HHNYNX34P0CR</RequestId>
    <HostId>W2Ui7jGmxqjbC4YDNBFlkwMOpvvOLiFfG3/+hKwkDf17y+RjG0DXgK5Ys6E9NqUChL3BAVDdTsM=</HostId>
  </Error>

• Even worse, if the input stream produces an empty (zero-byte) chunk, the lack of buffering results in the chunk being serialized as 0\r\n\r\n, which prematurely signals the end of the aws-chunked body and causes a malformed trailer error:

  <?xml version="1.0" encoding="UTF-8"?>
  <Error>
    <Code>MalformedTrailerError</Code>
    <Message>The request contained trailing data that was not well-formed or did not conform to our published schema.</Message>
    <RequestId>SQXM6H16K11AK9WQ</RequestId>
    <HostId>esc1L1ccaETEivpB096+T4kI0Q+bM5q7+Q7x6yKoaLe+jfK6FEQmR08EeHljLiqmBBB59a/Sqcmim9F2YQ04VCp7uDOwaf8GpaZyy4+Ondo=</HostId>
  </Error>

The above errors are fixed by explicitly passing the minimum chunk size to S3Client:

 const s3Client = new S3Client({
+  requestStreamBufferSize: 8 * 1024,
 });

Why is it that S3Client does not default requestStreamBufferSize to 8 * 1024, as above? This causes the aforementioned errors, which don't seem obvious (and I personally spent the last few days debugging this).

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-s3@3.934.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

Node.js v24.11.0

Reproduction Steps

See https://github.com/mxxk-examples/aws-sdk-js-v3-issue-7509.

Observed Behavior

Depending on the test, S3 errors out with InvalidChunkSizeError or MalformedTrailerError, as seen above.

Expected Behavior

S3Client properly buffers chunks smaller than 8 KiB (which also tolerates empty chunks from the input stream) without the caller needing to know about or supply requestStreamBufferSize.

Possible Solution

Make S3Client default requestStreamBufferSize to 8 * 1024 (8 KiB).

Additional Information/Context

Related issues:

• Streaming contents to S3 fails in latest typescript s3 sdk #7048

  This issue proposes a workaround, but does not address the core pain point (lack of sane default for requestStreamBufferSize).

• Support for minimum chunk size handling in streaming uploads (re: WHEN_REQUIRED) #6949

  This issue points to the need to provide requestStreamBufferSize, but it's not clear to me the design decision to not default requestStreamBufferSize.

[kuhe]

The SDK aims to serve a wide variety of customer use cases, some of which are balanced against each other.

The automatic buffering feature exists because I felt it may be inconvenient to implement in user code for users who need it.
But it remains optional rather than default because for a properly chunked input stream it is unnecessary overhead and would decrease throughput performance.

Node.js streams have a default buffer size that is sufficient for the 8kb chunk requirement, so the majority of use cases such as file read streams should not use this feature. What kind of input stream were you using that had smaller than 8kb and 0byte chunks?

[mxxk]

The automatic buffering feature exists because I felt it may be inconvenient to implement in user code for users who need it.
But it remains optional rather than default because for a properly chunked input stream it is unnecessary overhead and would decrease throughput performance.

Ah, I see... Thanks for clarifying. 🙂

Node.js streams have a default buffer size that is sufficient for the 8kb chunk requirement, so the majority of use cases such as file read streams should not use this feature. What kind of input stream were you using that had smaller than 8kb and 0byte chunks?

Good point... I initially encountered the 0\r\n\r\n premature termination error when I was using a makeshift test stream which produces a fixed number of null bytes:

repro.js

import fs from "node:fs";
import stream from "node:stream";

function getNullBytes(maxBytes) {
  return stream.Readable.from(
    (async function* readable() {
      // NOTE: The following alternative to `try / finally` throws `AbortError`
      // since the read stream is not fully consumed:
      //     await using devZero = fs.createReadStream("/dev/zero");
      const devZero = fs.createReadStream("/dev/zero");
      try {
        let bytes = 0;
        for await (const chunk of devZero) {
          const cap = Math.min(chunk.length, maxBytes - bytes);
          console.debug({
            maxBytes,
            bytes,
            chunkLength: chunk.length,
            cap,
          });
          if (cap === chunk.length) {
            yield chunk;
            bytes += chunk.length;
          } else {
            const subchunk = chunk.subarray(0, cap);
            yield subchunk;
            bytes += cap;
            break;
          }
        }
      } finally {
        devZero.destroy();
      }
    })(),
  );
}


await stream.promises.pipeline(
  getNullBytes(65536),
  new stream.Writable({
    write(chunk, encoding, callback) {
      console.debug("CHUNK", chunk, encoding);
      callback();
    },
  }),
);

But upon revisiting the above example, I noticed that getNullBytes does not check if it's about to yield an empty chunk, which happens in the above example because the number of requested bytes (maxBytes: 65536) is a multiple of Node.js' default highWaterMark: 65536. This can of course be prevented with a simple fix:

repro.js diff

diff --git a/dev/fd/0 b/dev/fd/3
--- a/repro.js
+++ b/repro.js
@@ -22,6 +22,9 @@ function getNullBytes(maxBytes) {
             yield chunk;
             bytes += chunk.length;
           } else {
+            if (!cap) {
+              break;
+            }
             const subchunk = chunk.subarray(0, cap);
             yield subchunk;
             bytes += cap;

Note

For the record, I should've written a better implementation of getNullBytes which does not need to open /dev/null, which I'm including here for posterity:

Better getNullBytes

function getByteStream(maxBytes, chunkSize) {
  return stream.Readable.from(
    (async function* generator() {
      let bytes = 0;
      while (bytes < maxBytes) {
        const chunk = Buffer.alloc(Math.min(chunkSize, maxBytes - bytes));
        yield chunk;
        bytes += chunk.length;
      }
    })(),
  );
}

Based on my current understanding of Node.js streams, and also what you're saying @kuhe, while it's possible to yield an empty chunk (or a small non-last chunk) in Node.js streams, this rarely occurs, and potentially, such a stream can be considered not "well-behaved".

---

In some cases, however, the input stream is entirely out of the control of the developer hooking it up to S3Client. A common example I have seen of this is where an HTTP response body is streamed into S3 to sync data. It is possible that for whatever reason, the source HTTP response returns a non-final chunk of data smaller than 8 KiB (for example, if the server generates some initial data, does more work, and serves the rest afterwards). This is demonstrated with chunkServer in https://github.com/mxxk-examples/aws-sdk-js-v3-issue-7509.

Another potentially problematic example I can think of is if fs.createReadStream() is being consumed, but the underlying filesystem can yield a non-final chunk smaller than 8 KiB. I'm not sure how possible this is if, say, the filesystem is NFS, or the underlying disk happens to be slow?

There's a common theme in the above two examples: the underlying source (network stream, or filesystem) might be slow. Again, I'm not an expert in Node.js streams, but from what I can tell, highWaterMark (default 65536) does not prevent chunks smaller than highWaterMark from being emitted; rather, it signals backpressure if more than highWaterMark amount of data is queued. It seems that Node.js streams with slow underlying sources can legitimately occur in the wild, in which case S3Client will mishandle them unless explicitly configured with requestStreamBufferSize: 8 * 1024 (or larger).

@kuhe given the above information, is your stance on this matter still the same? (That is, because we're dealing with edge case Node.js streams, it doesn't make sense to have the SDK unconditionally wrap the input stream meant for S3 in createBufferedReadable, as that would incur unnecessary overhead in the majority of cases.)

[kuhe]

I sympathize that the connection between the error message and the configuration property is hard to find, but my stance on the default setting (no modification of the user input stream) is the same.

if you:

• have control over both the input stream and S3 client
  • you decide whether you want automatic buffering or less overhead in the way of your transfer
• have control only of the input stream
  • you can ask the S3 client app owner to buffer the input or you can wrap the stream yourself
• have control only of the S3 client
  • enable stream buffering on the S3 client or wrap the input as your receive it

I don't think the stream is entirely out of one's control even if you aren't the producer of it. A pipe can be attached to either end of a stream by anyone who is along its path.

[mxxk]

I sympathize that the connection between the error message and the configuration property is hard to find, but my stance on the default setting (no modification of the user input stream) is the same.

Understood, and thanks for the clarification. Feel free to close this issue as not planned.

My hope is that Google will index this page and help future devs facing these unintuitive errors find a solution in this GitHub issue.

I don't think the stream is entirely out of one's control even if you aren't the producer of it. A pipe can be attached to either end of a stream by anyone who is along its path.

Right, but that would be reinventing the same utility already provided in this SDK. Either way—passing requestStreamBufferSize, or as you suggested, attaching your own buffering stream—the developer using this SDK needs to be aware of the caveat, which is not common knowledge. They would write code which seems to work correctly, until it doesn't—as it seems, many readable streams are built upon data sources with inherent variability outside of the dev's control (network speed, disk speed).

It’s a performance tradeoff, and while I may not agree with the decision, I do appreciate the explanation. It should help future users understand the behavior.

---

I'll open a separate issue for what seems like lack of proper backpressure handling in both streams, after more testing to confirm.