Configure access to a source: Amazon S3

You can set up access to an Amazon S3 bucket using either of two methods:

Supported regions

Storage Transfer Service supports the following Amazon S3 regions:
af-south-1
ap-east-1
ap-east-2
ap-northeast-1
ap-northeast-2
ap-northeast-3
ap-south-1
ap-south-2
ap-southeast-1
ap-southeast-2 		ap-southeast-3
ap-southeast-4
ap-southeast-5
ca-central-1
ca-west-1
eu-central-1
eu-central-2
eu-north-1
eu-south-1
eu-south-2 		eu-west-1
eu-west-2
eu-west-3
il-central-1
me-central-1
me-south-1
sa-east-1 		us-east-1
us-east-2
us-west-1
us-west-2
us-gov-east-1
us-gov-west-1
Transfers using the Google-managed private network support the following AWS regions:
ap-east-1
ap-northeast-1
ap-northeast-2
ap-northeast-3
ap-south-1
ap-south-2
ap-southeast-1
ca-central-1
ca-west-1
eu-central-1
eu-central-2
 eu-north-1
eu-south-1
eu-south-2
eu-west-1
eu-west-2
eu-west-3
us-east-1
us-east-2
us-west-1
us-west-2 		 

Required permissions

In order to use Storage Transfer Service to move data from an Amazon S3 bucket, your user account or federated identity role must have the appropriate permissions for the bucket:

Permission Description Use
s3:ListBucket Allows Storage Transfer Service to list objects in the bucket. Always required.
s3:GetObject Allows Storage Transfer Service to read objects in the bucket. Required if you are transferring the current version of all objects. If your manifest specifies an object version, use s3:GetObjectVersion instead.
s3:GetObjectVersion Allows Storage Transfer Service to read specific versions of objects in the bucket. Required if your manifest specifies an object version. Otherwise, use s3:GetObject.
s3:DeleteObject Allows Storage Transfer Service to delete objects in the bucket. Required if you set deleteObjectsFromSourceAfterTransfer to true.

Authenticate using access credentials

To use an access key ID and secret key to authenticate to AWS:

  1. Create an AWS Identity and Access Management (AWS IAM) user with a name that you can recognize, such as transfer-user.

  2. For AWS access type, select Access key - programmatic access.

  3. Grant one of the following roles to the user:

    • AmazonS3ReadOnlyAccess to provide read-only access to the source. This allows transfers but does not support deleting objects at source once the transfer is complete.
    • AmazonS3FullAccess if your transfer is configured to delete objects at source.

    • A custom role with the appropriate permissions from the Required permissions table. The JSON for the minimum permissions looks like the following example:

      Standard AWS regions 

      {
"Version": "2012-10-17",
"Statement": [
  {
    "Effect": "Allow",
    "Action": [
      "s3:GetObject",
      "s3:ListBucket"
    ],
    "Resource": [
      "arn:aws:s3:::S3_BUCKET_NAME/*",
      "arn:aws:s3:::S3_BUCKET_NAME"
    ]
  }
]
}

      AWS GovCloud regions 

      {
"Version": "2012-10-17",
"Statement": [
  {
    "Effect": "Allow",
    "Action": [
      "s3:GetObject",
      "s3:ListBucket"
    ],
    "Resource": [
      "arn:aws-us-gov:s3:::S3_BUCKET_NAME/*",
      "arn:aws-us-gov:s3:::S3_BUCKET_NAME"
    ]
  }
]
}

  4. Note the access key ID and secret access key when the user is successfully created.

How you pass the access key ID and secret access key to Storage Transfer Service depends on the interface you use to initiate the transfer.

Google Cloud console

Enter the values directly into the transfer job creation form.

See Create transfers to get started.

gcloud CLI

Create a JSON file with the following format:

{
  "accessKeyId": "AWS_ACCESS_KEY_ID",
  "secretAccessKey": "AWS_SECRET_ACCESS_KEY"
}

Pass the location of the file to the gcloud transfer jobs create command using the source-creds-file flag:

gcloud transfer jobs create s3://S3_BUCKET_NAME gs://GCS_BUCKET_NAME \
  --source-creds-file=PATH/TO/KEYFILE.JSON

REST API

Your transferSpec object must contain the key info as part of the awsS3DataSource object:

"transferSpec": {
  "awsS3DataSource": {
    "bucketName": "S3_BUCKET_NAME",
    "awsAccessKey": {
      "accessKeyId": "AWS_ACCESS_KEY_ID",
      "secretAccessKey": "AWS_SECRET_ACCESS_KEY"
    }
  },
  "gcsDataSink": {
    "bucketName": "GCS_BUCKET_NAME"
  }
}

Client libraries

See the examples in the Create transfers page.

Save your access credentials in Secret Manager

Secret Manager is a secure service that stores and manages sensitive data such as passwords. It uses strong encryption, role-based access control, and audit logging to protect your secrets.

Storage Transfer Service can leverage Secret Manager to protect your AWS access credentials. You load your credentials into Secret Manager, then pass the secret resource name to Storage Transfer Service.

Enable the API

Enable the Secret Manager API.