Create a Cloud Storage import topic

A Cloud Storage import topic lets you continuously ingest data from Cloud Storage into Pub/Sub. Then you can stream the data into any of the destinations that Pub/Sub supports. Pub/Sub automatically detects new objects added to the Cloud Storage bucket and ingests them.

Cloud Storage is a service for storing your objects in Google Cloud. An object is an immutable piece of data consisting of a file of any format. You store objects in containers called buckets. Buckets can also contain managed folders, which you use to provide expanded access to groups of objects with a shared name prefix.

For more information about Cloud Storage, see the Cloud Storage documentation.

For more information about import topics, see About import topics.

Before you begin

• Learn more about the Pub/Sub publish process.

• A Cloud Storage bucket must already exist before you create a Cloud Storage import topic. If you are using the console to create the import topic, the workflow lets you create a Cloud Storage bucket. For other configuration methods, see Create buckets.

• If applicable, ensure that the message storage policy of the Pub/Sub topic overlaps with the region where your Cloud Storage bucket is located. For more information, see Message storage policy is compliant with the bucket location.

• Some Google Cloud services have Google Cloud-managed service accounts that lets the services access your resources. These service accounts are known as service agents. Pub/Sub creates and maintains a service account for each project in the format service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com. Configure the required roles and permissions on the Pub/Sub service account to manage Cloud Storage import topics including the following:

  • Grant the Pub/Sub publisher role (roles/pubsub.publisher) to the Pub/Sub service account. This service account is going to publish to the import topic. To grant this role, you require a user account with the Pub/Sub Admin role (roles/pubsub.admin). For more information, see Add the Pub/Sub publisher role to the Pub/Sub service account.

  • Grant Cloud Storage permissions to the Pub/Sub service account. To grant these permissions, you require a user account with the Storage Admin role (roles/storage.admin). For more information, see Assign Cloud Storage roles to the Pub/Sub service account.

Required roles and permissions

To get the permissions that you need to create and manage a Cloud Storage import topic, ask your administrator to grant you the Pub/Sub Editor (roles/pubsub.editor) IAM role on your topic or project. For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the permissions required to create and manage a Cloud Storage import topic. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to create and manage a Cloud Storage import topic:

• Create an import topic: pubsub.topics.create
• Delete an import topic: pubsub.topics.delete
• Get an import topic: pubsub.topics.get
• List an import topic: pubsub.topics.list
• Publish to an import topic: pubsub.topics.publish
• Update an import topic: pubsub.topics.update
• Get the IAM policy for an import topic: pubsub.topics.getIamPolicy
• Configure the IAM policy for an import topic: pubsub.topics.setIamPolicy

You might also be able to get these permissions with custom roles or other predefined roles.

You can configure access control at the project level and the individual resource level.

Message storage policy is compliant with the bucket location

The message storage policy of the Pub/Sub topic must overlap with the regions where your Cloud Storage bucket is located. This policy dictates where Pub/Sub is allowed to store your message data.

• For buckets with location type as region: The policy must include that specific region. For example, if your bucket is in the us-central1 region, the message storage policy must also include us-central1.

• For buckets with location type as dual-region or multi-region: The policy must include at least one region within the dual-region or multi-region location. For example, if your bucket is in the US multi-region, the message storage policy could include us-central1, us-east1, or any other region within the US multi-region.

  If the policy doesn't include the bucket's region, topic creation fails. For example, if your bucket is in europe-west1 and your message storage policy only includes asia-east1, you'll receive an error.