Configure log buckets

This document describes how to create and manage Cloud Logging buckets using the Google Cloud console, the Google Cloud CLI, and the Logging API. It also provides instructions for creating and managing log buckets at the Google Cloud project level. You can't create log buckets at the folder or organization level; however, Cloud Logging automatically creates _Default and _Required log buckets at the folder and organization level for you.

You can upgrade log buckets to use Observability Analytics. Observability Analytics lets you run SQL queries on your log data, helping you troubleshoot application, security, and networking issues.

To use BigQuery to analyze your log data, you have two choices:

Upgrade a log bucket to use Observability Analytics and then create a linked BigQuery dataset. In this scenario, Logging stores your log data but BigQuery can read the log data.
Export your log entries to BigQuery. In this scenario, you must create a sink, BigQuery stores and manages the data, and you can use partitioned tables.

When your log data is available to BigQuery, you can join your log data with other data stored in BigQuery, and you can access also use tools such as Data Studio and Looker.

For a conceptual overview of buckets, see Store log entries.

This document doesn't describe how to create a log bucket that uses a customer-managed encryption key (CMEK). If you want to configure CMEK, then see Configure CMEK for logs storage.

Before you begin

Configure your project, your IAM roles, and select the interface that you plan to use.

Configure your project and roles

To get started with buckets, do the following:

Configure your Google Cloud project:
1. Verify that billing is enabled for your Google Cloud project.
2. To get the permissions that you need to create, upgrade, and link a log bucket, ask your administrator to grant you the Logs Configuration Writer (roles/logging.configWriter) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.
  
  You might also be able to get the required permissions through custom roles or other predefined roles.
  
  For the full list of permissions and roles, see Access control with IAM.
3. Optional: To use BigQuery to view the data stored in a log bucket, do the following:
  1. Make sure that the BigQuery API is enabled. You can verify that the API is enabled by listing available services.
  2. Your Identity and Access Management role includes the permissions that let you create a linked BigQuery dataset. For more information, see Permissions for linked BigQuery datasets.
Understand the supported regions in which you can store your logs.
If you plan to use the Google Cloud CLI or Cloud Logging API to create or manage your log buckets, then understand the LogBucket formatting requirements.

Select the interface you plan to use

Google Cloud console

When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

gcloud

In the Google Cloud console, activate Cloud Shell.

Activate Cloud Shell

At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

REST

To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

For more information, see Authenticate for using REST in the Google Cloud authentication documentation.

Create a bucket

You can create a maximum of 100 buckets per Google Cloud project. You can't create log buckets in folders or organizations.

To create a user-defined log bucket for your Google Cloud project, do the following:

Google Cloud console

To create a log bucket in your Google Cloud project, do the following:

In the Google Cloud console, go to the Logs Storage page:
Go to Logs Storage

If you use the search bar to find this page, then select the result whose subheading is Logging.
Click Create log bucket.
Enter a Name and Description for your bucket.
Optional: Upgrade your bucket to use Observability Analytics.

Note: After a log bucket has been upgraded to use Observability Analytics, you can't reconfigure the log bucket to change or remove the use of Observability Analytics.
1. Select Upgrade to use Observability Analytics.
  When you upgrade a bucket to use Observability Analytics, you can query your logs in the Observability Analytics page by using SQL queries. You can also continue to view your logs by using the Logs Explorer.
2. Optional: To view your logs in BigQuery, select Create a new BigQuery dataset that links to this bucket and enter a unique dataset name.
  When you select this option, BigQuery can read the data stored in your log bucket. You can now query in the BigQuery interface where you can join your log data, and also access data from other tools like Data Studio and Looker.
To select the storage region for your logs, click the Select log bucket region menu and select a region.

Note: Log buckets in the global region or in multi-regions don't offer added resiliency compared to log buckets created in single regions like us-west1.

Note: After you create your log bucket, you can't change your bucket's region. If you need a bucket in a different region, you must create a new bucket in that region, redirect the appropriate sinks to the new bucket, and then delete the old bucket.
Optional: To set a custom retention period for the logs in the bucket, click Next.

In the Retention period field, enter the number of days, between 1 day and 3650 days, that you want Cloud Logging to retain your logs. If you don't customize the retention period, the default is 30 days.

You can also update the retention period for your log bucket after you create it.
Click Create bucket.

After the log bucket is created, Logging upgrades the bucket and creates the dataset link, if these options were selected.

It might take a moment for these steps to complete.

gcloud

To only create a log bucket, run the gcloud logging buckets create command. If you want to upgrade the log bucket to use Observability Analytics, then include the --enable-analytics and --async flags, and make sure that you set the variable LOCATION to a supported region:

gcloud logging buckets create BUCKET_ID --location=LOCATION --enable-analytics --async OPTIONAL_FLAGS

The flag --async forces the command to be asynchronous. The return of an asynchronous method is an Operation object, it contains information about the progress of the method. When the method completes, the Operation object contains the status. For more information, see Asynchronous API methods.

If you don't want to upgrade the log bucket to use Observability Analytics, then omit the --enable-analytics and --async flags.

For example, if you want to create a bucket with the BUCKET_ID my-bucket in the global region, your command would look like the following:

gcloud logging buckets create my-bucket --location global --description "My first bucket"

For example, to create a bucket with the BUCKET_ID my-upgraded-bucket in the global location, and then upgrade the log bucket to use Observability Analytics, your command would look like the following:

gcloud logging buckets create my-upgraded-bucket --location global \
      --description "My first upgraded bucket" \
      --enable-analytics --retention-days=45

REST

To create a bucket, use the projects.locations.buckets.create or the projects.locations.buckets.createAsync method. Prepare the arguments to the method as follows:

Set the parent parameter to be the resource in which to create the bucket: projects/PROJECT_ID/locations/LOCATION

The variable LOCATION refers to the region in which you want your logs to be stored.

For example, if you want to create a bucket for project my-project in the in the global region, your parent parameter would look like this: projects/my-project/locations/global
Set the bucketId parameter; for example, my-bucket.
Do one of the following:
- To create a log bucket and then upgrade the log bucket to use Observability Analytics:
  1. Set the LogBucket.analyticsEnabled boolean to true.
    
    Note: After a log bucket has been upgraded to use Observability Analytics, you can't reconfigure the log bucket to change or remove the use of Observability Analytics.
  2. Call the asynchronous method projects.locations.buckets.createAsync to create the bucket.
    
    The response to the asynchronous methods is an Operation object. This object contains information about the progress of the method. When the method completes, the Operation object contains the status. For more information, see Asynchronous API methods.
    
    The createAsync method takes several minutes to complete. This method method doesn't generate an error message or fail when the analyticsEnabled boolean is set to true.

Configure log buckets Stay organized with collections Save and categorize content based on your preferences.

Before you begin

Configure your project and roles

Select the interface you plan to use

Google Cloud console

gcloud

REST

Create a bucket

Google Cloud console

gcloud

REST

Configure log buckets