s3-proxy

Proxy that transparently authenticates read-only S3 requests

For read-only ListObject and GetObject API requests for index.html files, the proxy will retrieve the data from the backend and return it directly.

For other GetObject API requests, the proxy will issue a 307 temporary redirect to a presigned URL.

Configuration

Environment variable	Type	Secret	Required	Description
`AWS_ACCESS_KEY_ID`	string	no	yes	Access key of the AWS service account that has GetObject and ListObject permissions
`AWS_SECRET_ACCESS_KEY`	string	yes	yes	Secret key of the AWS service account that has GetObject and ListObject permissions
`AWS_DEFAULT_REGION`	string	no	no	AWS region, defaults to `us-east-1`
`AWS_ENDPOINT_HOST`	string	no	no	host name of the backend server used for proxied requests, defaults to `s3.<region>.amazonaws.com`
`AWS_ENDPOINT_HOST_EXTERNAL`	string	no	no	host name of the backend server used for redirects, defaults to `<host>`
`AWS_ENDPOINT_SCHEME`	string	no	no	schema to use for the backend server, defaults to `https`

Caveats

While the proxy is able to forward arbitrary S3 API requests, it does not present a fully S3-compliant interface to clients.

In particular, aws s3 sync and minio-client mirror do not work with the proxy, as they do not support the 307 temporary redirects.

As a workaround, you can use a command like the following to sync data from the proxy to a local directory:

ENDPOINT_URL=https://s3-proxy
BUCKET_NAME=bucket-name
PREFIX=some/prefix
OUTPUT_DIRECTORY=/tmp/s3-data
aws \
  --endpoint-url "${ENDPOINT_URL}" --no-sign-request \
  s3api list-objects-v2 --bucket "${BUCKET_NAME}" --prefix "${PREFIX}" \
  | jq --raw-output '.Contents[].Key' \
  | xargs --max-procs 256 -I{} \
  curl --location --no-progress-meter --create-dirs --output "${OUTPUT_DIRECTORY}/{}" \
  "${ENDPOINT_URL}/${BUCKET_NAME}/{}"