Transfer specific files or objects using a manifest

Storage Transfer Service supports the transfer of specific files or objects, which are specified using a manifest. A manifest is a CSV file, uploaded to Cloud Storage, that contains a list of files or objects for Storage Transfer Service to act upon.

A manifest can be used for the following transfers:

• From AWS S3, S3-compatible storage, Azure Blobstore, or Cloud Storage to a Cloud Storage bucket.

• From a file system to a Cloud Storage bucket.

• From a Cloud Storage bucket to a file system.

• Between two file systems.

• From a publicly-accessible HTTP/HTTPS source to a Cloud Storage bucket. Follow the instructions in Create a URL list as the manifest format is unique for URL lists.

Create a manifest

Manifest files have the following requirements:

• Manifests must be formatted as CSV.
• They can contain any UTF-8 characters.
• The first column must be a filename or object name. The name is relative to the root path or the bucket and folder specified in the transfer job; see File system transfers and Object storage transfers for details.
• Manifest files do not support wildcards. Folder names without a file or object name are not supported.
• If a file or object name contains a comma, the name must be enclosed in double-quotes. For example, "doe,john.txt".
• There is no limit to the size of the manifest file.

We recommend testing your transfer with a small subset of files or objects to avoid unnecessary API calls due to configuration errors.

You can monitor the status of file transfers from the Transfer Jobs page. Files or objects that fail to transfer are listed in the transfer logs.

File system transfers

To create a manifest of files on a file system, create a CSV file with a single column containing the file paths relative to the root directory specified in the transfer job creation.

For example, to transfer the following file system files:

Your manifest should look like the following example:

dir1/subdir1/file1.txt
file2.txt
dir2/subdir1/file3.txt

Save the manifest file with any filename, and a .csv extension.

Object storage transfers

To create a manifest of objects, create a CSV file whose first column contains the object names relative to the bucket name and path specified in the transfer job creation. All objects must be in the same bucket.

You can also specify an optional second column with the Cloud Storage generation number of the specific version to transfer.

For example, you may wish to transfer the following objects:

Your manifest should look like the following example:

object1.pdf,1664826685911832
object2.pdf
object3.pdf,1664826610699837

Save the manifest file with any filename, and a .csv extension.

HTTP/HTTPS transfers

To transfer specific files from an HTTP or HTTPS source, refer to the instructions in Create a URL list.

Publish the manifest

Once you've created the manifest, you must make it available to Storage Transfer Service. Storage Transfer Service can access the file in a Cloud Storage bucket, or on your file system.

Upload the manifest to Cloud Storage

You can store the manifest file in any Cloud Storage bucket.

The service agent running the transfer must have storage.objects.get permission for the bucket containing the manifest. See Grant the required permissions for instructions on finding the service agent ID, and granting permissions to that service agent on a bucket.

For instructions on uploading the manifest to a bucket, see Upload objects in the Cloud Storage documentation.

For example, to use the gcloud CLI to upload a file to Cloud Storage, use the gcloud storage cp command:

gcloud storage cp MANIFEST.CSV gs://DESTINATION_BUCKET_NAME/

Replace the following:

• MANIFEST.CSV is the local path to your manifest file. For example, Desktop/manifest01.csv.

• DESTINATION_BUCKET_NAME is the name of the bucket to which you are uploading your object. For example, my-bucket.

If successful, the response looks like the following example:

Completed files 1/1 | 164.3kiB/164.3kiB

You can encrypt a manifest using customer-managed Cloud KMS encryption keys. In this case, ensure that any service accounts accessing the manifest are assigned the applicable encryption keys. Customer-supplied keys are not supported.

Store the manifest on a file system

You can store the manifest file on your source or destination file system.

The location of the file must be accessible to the transfer agents. If you restrict directory access for your agents, make sure the manifest file is located within a mounted directory.

Start a transfer

Do not modify the manifest file until a transfer operation completes. We recommend that you lock the manifest file when a transfer is taking place.

gcloud transfer jobs create SOURCE