Uploads fail due to open issue in s3transferpackage

[andybak]

boto/s3transfer#327

The issue is new so there's no fix yet but keep an eye on it. I can't see an obvious workaround for us (but I'm also not sure when this started happening?)

This is what you see in our logs:

    return self.meta.client.upload_fileobj(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/boto3/s3/inject.py", line 642, in upload_fileobj
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/s3transfer/futures.py", line 264, in result
    raise self._exception
  File "/venv/lib/python3.11/site-packages/s3transfer/tasks.py", line 135, in __call__
    return self._execute_main(kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/s3transfer/tasks.py", line 158, in _execute_main
    return_value = self._main(**kwargs)
                   ^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/s3transfer/upload.py", line 796, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/venv/lib/python3.11/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)

botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unsupported header 'x-amz-sdk-checksum-algorithm' received for this API call.

[andybak]

On a side note - the site also froze up for a while shortly after. Could this be related?

[2025-01-18 10:38:39,521] INFO:huey:Worker-1:Executing icosa.tasks.queue_finalize_asset: ce20a784-a4f4-4f2c-90d9-add856b3a42f
[2025-01-18 10:38:39,622] INFO:huey:Worker-1:icosa.tasks.queue_finalize_asset: ce20a784-a4f4-4f2c-90d9-add856b3a42f executed in 0.099s
[2025-01-18 11:48:18 +0000] [17] [CRITICAL] WORKER TIMEOUT (pid:25)
[2025-01-18 11:48:18 +0000] [25] [INFO] Worker exiting (pid: 25)
[2025-01-18 11:48:19 +0000] [95] [INFO] Booting worker with pid: 95

[andybak]

The above issue is reported as closed/fixed now.

[jamesrampton]

I've updated b2sdk to latest which includes the fixed version of s3transfer.

Let me know if you have any issues. I'll test this myself too.

[jamesrampton]

This should actually be fixed now @andybak. Using the latest libraries was not enough. Turns out B2 doesn't have complete parity with S3 in these ways: https://www.backblaze.com/docs/cloud-storage-s3-compatible-api#unsupported-features

I've special-cased B2 storage configuration with some esoteric config to squash adding the x-amz-checksum-* headers.