s3-proxy

Proxy that transparently authenticates read-only S3 requests

The s3-proxy provides authenticated read-only access to private S3 buckets without exposing AWS credentials.

How It Works

The proxy uses nginx with the njs JavaScript module to handle AWS Signature V4 authentication. When a request arrives, the proxy behaves differently based on the requested path:

For index.html files and bucket listings:

  • The proxy retrieves the content from S3 directly
  • Returns the data to the client through the proxy
  • These must be proxied (not redirected) because the HTML contains relative links that would break if served directly from S3

For all other files (artifacts, logs, JSON, etc.):

  • The proxy generates a temporary presigned URL (valid for 5 minutes)
  • Returns an HTTP 307 redirect to the presigned S3 URL
  • The client downloads directly from S3 using the presigned URL
  • Reduces bandwidth and load on the proxy infrastructure

This approach provides authenticated access to private S3 buckets while minimizing proxy resource usage for large file downloads.

Directory Browsing

The s3-proxy serves index.html files generated by s3-bucket-listing to provide directory browsing. These static index pages list bucket contents and are served through the proxy (not redirected).

Configuration

Environment variable Type Secret Required Description
AWS_ACCESS_KEY_ID string no yes Access key of the AWS service account that has GetObject and ListObject permissions
AWS_SECRET_ACCESS_KEY string yes yes Secret key of the AWS service account that has GetObject and ListObject permissions
AWS_DEFAULT_REGION string no no AWS region, defaults to us-east-1
AWS_ENDPOINT_HOST string no no host name of the backend server used for proxied requests, defaults to s3.<region>.amazonaws.com
AWS_ENDPOINT_HOST_EXTERNAL string no no host name of the backend server used for redirects, defaults to <host>
AWS_ENDPOINT_SCHEME string no no schema to use for the backend server, defaults to https

Caveats

While the proxy is able to forward arbitrary S3 API requests, it does not present a fully S3-compliant interface to clients.

In particular, aws s3 sync and minio-client mirror do not work with the proxy, as they do not support the 307 temporary redirects.

As a workaround, you can use a command like the following to sync data from the proxy to a local directory:

ENDPOINT_URL=https://s3-proxy
BUCKET_NAME=bucket-name
PREFIX=some/prefix
OUTPUT_DIRECTORY=/tmp/s3-data
aws \
  --endpoint-url "${ENDPOINT_URL}" --no-sign-request \
  s3api list-objects-v2 --bucket "${BUCKET_NAME}" --prefix "${PREFIX}" \
  | jq --raw-output '.Contents[].Key' \
  | xargs --max-procs 256 -I{} \
  curl --location --no-progress-meter --create-dirs --output "${OUTPUT_DIRECTORY}/{}" \
  "${ENDPOINT_URL}/${BUCKET_NAME}/{}"