S3 Backend
The S3 backend redirects object requests to another S3 endpoint. The IAM accounts are still handled within the versitygw, and all requests to the external S3 service use the provided account credentials in the backend configuration.
./versitygw --access rootuser --secret rootpass s3 --access backenduser --secret backendpass --endpoint http://my.s3server.com
The --access/--secret after the s3 defines the credentials to use for accessing the backend S3 system.
The S3 API requests are validated on the frontend and then relayed to the backend. So the backend must support any incoming service request to versitygw. All backend requests are regenerated using the configured backend credentials, and requests to the backend are signed using standard v4 authentication.
Some on-premise S3 systems may not allow certain APIs such as CreateBucket, etc. These would also not be allowed through the gateway.
versitygw [global options] s3 [command options]
--access value, -a value s3 proxy server access key id
--secret value, -s value s3 proxy server secret access key
--endpoint value s3 service endpoint, default AWS if not specified
--region value s3 service region, default 'us-east-1' if not specified (default: "us-east-1")
--disable-checksum disable gateway to server object checksums (default: false)
--ssl-skip-verify skip ssl cert verification for s3 service (default: false)
--debug output extra debug tracing (default: false)
--help, -h show help
The --access/--secret/--region/--endpoint should be configured based on the required access to the backend S3 system. All access to the backend system will use this single account.
The --disable-checksum option currently has no effect on PutObject/PutPart due to #345. All streaming data requests have content checksums disabled.
The --ssl-skip-verify option is sometimes needed for on-premise S3 object storage systems that use self signed SSL certs.
The internal versitygw IAM service is unrelated to the backend storage. All frontend accounts are still functional based on the configured IAM service. Multi-user is still functional with the s3 backend service, but all access to the backend is done solely with the configured single backend access account.
- The original use case was for a performance comparison of direct S3 access vs through the gateway with a client side benchmark utility.
- Another use case might be for local gateway account management with a single backend storage system account. This setup lets the main administrator access all the stored data, while allowing multiple users to securely access and use the data through the gateway.
AWS introduced new data integrity protections, and made this the default behavior is AWS supported client SDKs (including the Go SDK that versitygw makes use of when connecting to backend S3 services). See the blog post announcement here: https://aws.amazon.com/blogs/aws/introducing-default-data-integrity-protections-for-new-objects-in-amazon-s3/. This may not be compatible with S3 servers that do not yet support this feature. In this case you may see errors similar to the following:
https (backend):
SDK 2025/03/01 18:23:40 DEBUG request failed with unretryable error https response error StatusCode: 400, RequestID: ac150022:18fc90c5a22:f12cf:4, HostID: 589643e4d6ff0296065ad187ee47787ca9cc70eae5cf0091e2d2e9298b1aa2d1, api error XAmzContentSHA256Mismatch: The Content-SHA256 you specified did not match what we received
http (backend):
SDK 2025/03/01 18:25:11 DEBUG request failed with unretryable error compute input header checksum failed, unseekable stream is not supported without TLS and trailing checksum
2025/03/01 18:25:11 Internal Error, operation error S3: PutObject, compute input header checksum failed, unseekable stream is not supported without TLS and trailing checksum
To fix this, the following environment variable can be set to tell the SDK to use the older behavior. Even with this set, versitygw will still support the data integrity features when clients access versitygw with the feature enabled in the request. This setting is only adjusting behavior of versitygw communicating with the backend S3 service.
export AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED
