Fully migrate from Amazon S3 to Cloud Storage

This page describes how to fully migrate from Amazon Simple Storage Service (Amazon S3) to Cloud Storage for users sending requests using an API. After you fully migrate, you can use all the features of Cloud Storage, including multiple projects and OAuth 2.0 for authentication.

If you want to get started with Cloud Storage quickly, you can choose a simple migration, which requires just a few changes to the tools and libraries you're using with Amazon S3.

Migrate from Amazon S3 to Cloud Storage

To fully migrate from Amazon S3 to Cloud Storage, you need to complete the following steps:

  • Change any existing x-amz-* headers to corresponding x-goog-* headers.
  • Change AWS Access Control List (ACL) XML to the corresponding Cloud Storage ACL XML (see Creating and managing access control lists).
  • Set the x-goog-project-id header in your requests.

  • Get set up to use OAuth 2.0 authentication. Using OAuth 2.0 means that your Authorization header looks like this:

    Authorization: Bearer OAUTH2_TOKEN

    OAuth 2.0 relies on SSL for security instead of requiring your application to do cryptographic signing directly and is easier to implement. With OAuth, your application can request access to data associated with a user's account, and access can be scoped to several levels, including read-only, read-write, and full-control. For more information, see Cloud Storage OAuth 2.0 scopes and Accessing data on a user's behalf.

Access control

This section shows a few examples of access control to help you migrate from Amazon S3 to Cloud Storage. For an overview of access control in Cloud Storage, see Access Control.

In Cloud Storage, there are several ways to apply ACLs to buckets and objects (see Creating and managing access control lists). Two of the ways you specify ACLs are analogous to what you do in Amazon S3:

  • The acl query string parameter lets you apply ACLs for specific scopes.
  • The x-goog-acl request header lets you apply predefined ACLs, which are sometimes known as canned ACLs.

Using the acl query string parameter

You can use the acl query string parameter for a Cloud Storage request exactly the same way you would use it for an Amazon S3 request. The acl parameter is used in conjunction with the PUT method to apply ACLs to the following: an existing object, an existing bucket, or a bucket you are creating. When you use the acl query string parameter in a PUT request, you must attach an XML document (using Cloud Storage ACL syntax) to the body of your request. The XML document contains the individual ACL entries that you want to apply to the bucket or object.

The following example shows a PUT request to Amazon S3 that uses the acl query string parameter. ACLs are defined in an XML document sent in the request body. The PUT request changes the ACLs on an object named europe/france/paris.jpg that is in a bucket named my-travel-maps. The ACL grants jeffersonloveshiking@gmail.com FULL_CONTROL permission.

PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.s3.amazonaws.com
Date: Wed, 06 Nov 2013 19:28:18 GMT
Content-Length: 598
Content-Type: application/xml
Authorization: AWS4-HMAC-SHA256 Credential=AWS-ACCESS-KEY/20131106/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;date;host, Signature=4c45f25bb679fdab0de5a287625d6a143414728d93c9aeb9f4cc91c33a1c45fg

<?xml version='1.0' encoding='utf-8'?>
<AccessControlPolicy>
  <Owner>
    <ID>5a6557ba40f7c86496ffceae789fcd888abc1b62a7149873a0fe12c0f60a7d95</ID>
    <DisplayName>ownerEmail@example.com</DisplayName>
  </Owner>
  <AccessControlList>
    <Grant>
      <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
        <ID>fd447671d60b979f78ee6fcec7b22afc80e6b26a4db16eed01afb8064047949b</ID>
        <DisplayName>jeffersonloveshiking@gmail.com</DisplayName>
      </Grantee>
      <Permission>FULL_CONTROL</Permission>
    </Grant>
  </AccessControlList>
</AccessControlPolicy>

Here is the same request to Cloud Storage:

PUT europe/france/paris.jpg?acl HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Wed, 06 Nov 2013 19:37:33 GMT
Content-Length: 268
Content-Type: application/xml
Authorization: Bearer ya29.AHES6ZRVmB7fkLtd1XTmq6mo0S1wqZZi3-Lh_s-6Uw7p8vtgSwg

<?xml version='1.0' encoding='utf-8'?>
<AccessControlList>
  <Entries>
  <Entry>
    <Permission>FULL_CONTROL</Permission>
    <Scope type="UserByEmail">
      <EmailAddress>jeffersonloveshiking@gmail.com</EmailAddress>
    </Scope>
  </Entry>
  </Entries>
</AccessControlList>

Note that Cloud Storage does not require an <Owner/> element in the ACL XML document. For more information, see Bucket and object ownership.

You can also retrieve bucket and object ACLs by using the acl query string parameter with the GET method. The ACLs are described in an XML document, which is attached to the body of the response. You must have FULL_CONTROL permission to apply or retrieve ACLs on an object or bucket.

Apply ACLs with an extension request header

You can use the x-goog-acl header in a Cloud Storage request to apply predefined ACLs to buckets and objects exactly the same way you would use the x-amz-acl header in an Amazon S3 request. You typically use the x-goog-acl (x-amz-acl) header to apply a predefined ACL to a bucket or object when you are creating or uploading the bucket or object. The Cloud Storage predefined ACLs are similar to Amazon S3 Canned ACLs, including private, public-read, public-read-write, as well as others. For a list of Cloud Storage predefined ACLs, see Predefined ACLs.

The following example shows a PUT Object request that applies the public-read ACL to an object named europe/france/paris.jpg that is being uploaded into a bucket named my-travel-maps in Amazon S3.

PUT europe/france/paris.jpg HTTP/1.1
Host: my-travel-maps.s3.amazonaws.com
Date: Wed, 06 Nov 2013 20:48:42 GMT
Content-Length: 888814
Content-Type: image/jpg
x-amz-acl: public-read
Authorization: AWS4-HMAC-SHA256 Credential=AWS-ACCESS-KEY/20131106/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;date;host, Signature=808150c37dbd1b425b2398421d6fc3dd6d4942dfaae9e519fd5835aa62fd62ab

<888814 bytes in entity body>

Here is the same request to Cloud Storage:

PUT europe/france/paris.jpg HTTP/1.1
Host: my-travel-maps.storage.googleapis.com
Date: Wed, 06 Nov 2013 20:49:57 GMT
Content-Length: 888814
Content-Type: image/jpg
x-goog-acl: public-read
Authorization: Bearer ya29.AHES6ZRVmB7fkLtd1XTmq6mo0S1wqZZi3-Lh_s-6Uw7p8vtgSwg

<888814 bytes in entity body>

You can also use the x-goog-acl header to apply a predefined ACL to an existing bucket or object. To do this, include the acl query string parameter in your request but don't include an XML document in your request. Applying a predefined ACL to an existing object or bucket is useful if you want to change from one predefined ACL to another, or you want to update custom ACLs to a predefined ACL. For example, the following PUT Object request applies the predefined ACL private to an object named