Manage transfer agents

Storage Transfer Service agents are applications running inside an Open Container Initiative (OCI) container, that coordinate with Storage Transfer Service for transfers involving file systems or S3-compatible storage.

By default, Storage Transfer Service uses Docker to build and run OCI containers. Storage Transfer Service also supports Podman for container management; you must install your agents using the podman run command in order to use Podman.

If your transfer does not involve a file system or S3-compatible storage, you do not need to set up agents.

This document describes how to administer transfer agents on your servers.

Overview

  • Agent processes are dynamic. While you are running a transfer, you can add agents to increase performance. Newly started agents join the assigned agent pool and perform work from existing transfers. You can use this to adjust how many agents are running, or to adapt transfer performance to changing transfer demand.

  • Agent processes are a fault-tolerant collective. If one agent stops running, the remaining agents continue to do work. If all of your agents stop, when you add new agents the transfer resumes where the agents stopped. This enables you to avoid monitoring agents, retrying transfers, or implementing recovery logic. You can patch, move, and dynamically scale your agent pools without transfer downtime by coordinating agents with Google Kubernetes Engine.

    For example, you submit two transfers while two agents are running. If one of the agents stops due to a machine reboot or operating system patch, the remaining agent continues working. The two transfers are still running, but slower since a single agent is moving data. If the remaining agent also stops, then all transfers stop making progress, since there are no agents running. When you restart the agent processes, the transfers resume where they left off.

  • Agent processes belong to a pool. They collectively move your data in parallel. Because of this, all agents within a pool must have the same access to all data sources that you want to transfer.

    For example, if you are transferring data from a particular file system, you must mount the file system to every machine that is hosting agents in your agent pool. If some agents in your pool can reach a data source and others can't, transfers from that data source won't succeed.

Before you begin

Before configuring your transfers, make sure you have configured access: for users and service accounts.

If you'll be using gcloud commands, install the gcloud CLI.

Install and run transfer agents

We recommend installing a minimum of three agents per agent pool, ideally on separate machines. For more information about determining how many agents to run, see Maximizing transfer agent performance.

Don't include sensitive information such as personally identifiable information (PII) or security data in your agent ID prefix. Resource names may be propagated to the names of other Google Cloud resources and may be exposed to Google-internal systems outside of your project.

To install and run transfer agents:

Google Cloud console

  1. In the Google Cloud console, go to the Agent pools page.

    Go to Agent pools

  2. Select the agent pool to which to add the new agent.

  3. Click Install agent.

  4. Follow the instructions to install and run the agent.

    For more information about the agent's command-line options, see Agent command-line options.

gcloud CLI

To install one or more agents using the gcloud CLI, run gcloud transfer agents install:

gcloud transfer agents install --pool=POOL_NAME --count=NUM_AGENTS \
  --mount-directories=MOUNT_DIRECTORIES

The tool walks you through any required steps to install the agent(s). This command installs NUM_AGENTS agent(s) on your machine, mapped to the pool name specified as POOL_NAME, and authenticates the agent using your gcloud credentials. The pool name must exist, or an error is returned.

The --mount-directories flag is optional but is strongly recommended. Its value is a comma-separated list of directories on the file system to which to grant the agent access. Omitting this flag mounts the entire file system to the agent container. See the gcloud reference for more details.

S3-compatible sources

When installing agents to use with an S3-compatible source, you must provide AWS credentials either as environment variables as the values of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or stored as default credentials in your system's configuration files.

export AWS_ACCESS_KEY_ID=ID
export AWS_SECRET_ACCESS_KEY=SECRET
gcloud transfer agents install --pool=POOL_NAME \
  --creds-file=/relative/path/to/service-account-key.json

Use a service account key

To run agents using a service account key, use the --creds-file option: