S3 Provider¶

The S3 provider uses S3 to store commits remotely in a S3 bucket. Each commit is stored as an tar archive, with the commit metadata attached as object metadata. The URI format is:

s3://<bucket>/<key>

Commits will be created at <key>/<commitid>/<archive>.tar.gz. The commit metadata will be stored at the <commitid> level. In addition, the commits are stored in file named titan at the root of the repository. This makes it more efficient to list commits, and allows the S3 Web Provider to pull data.

The AWS credentials are pulled using the default AWS credential chain at the time you do the push or pull operation. So you must have the AWS_* environment variables set, or use your ~/.aws configuration. Because the S3 provider uses the standard AWS SDK, all variations of credentials should be supported, including specifying a profile with AWS_PROFILE or using MFA (multi factor authentication) session tokens. To pull a commit, you will need s3:GetObject permissions. To push a commit, you will need s3:PutObject permissions.

Although not recommended, it is possible to pass accessKey, secretKey, and region to the S3 Provider during runtime through CLI parameters:

s3://<bucket>/<key> -p accessKey=value -p secretKey=value -p region=value

The S3 provider relies on basic AWS APIs to implement its functionality, and as such has limited scalability. For example, all commits are stored in a single metadata file, and there are no provisions for keeping that metadata consistent in the face of simultaneous pushes. It should only be used for storing relatively small numbers of commits. Improving this will require a new provider that includes a robust metadata layer on top of the base S3 functionality.

accessKey=aws_access_key_id         AWS Access Key ID
secretKey=aws_secret_access_key     AWS Secret Access Key
region=region                       AWS Region