Create a linked BigQuery dataset

You can create a linked BigQuery dataset on your observability dataset, which lets you use BigQuery services to query your trace data. You can then join trace data with business datasets and analyze spans by using BigQuery Studio or the BigQuery API.

You don't need a linked BigQuery dataset to analyze your trace data with SQL if you use Observability Analytics. You can also join your trace and log data when you use Observability Analytics. For more information, see Query and analyze traces with SQL and Query and analyze logs with Observability Analytics.

When you need a linked dataset

You need a linked BigQuery dataset when you want to do any of the following:

Create a linked BigQuery dataset

This section describes how to create a linked BigQuery dataset on an observability dataset, which lets you use BigQuery services to query your trace data. Each observability dataset supports one linked BigQuery dataset.

When you create a linked dataset, you initiate a long-running operation. Several actions occur as part of that operation:

  • Audit logs record the request to create a link and the completion of the operation.

  • When the Monitoring Service Account doesn't exist, Google Cloud Observability creates it.

    The Cloud Monitoring API must be enabled for this service account to be created.

  • Google Cloud Observability requires a service account before it can create a linked dataset. When necessary, a service account is created:

  • An audit log records the request from the service agent manager to grant the Monitoring Service Account the IAM role of Monitoring Service Agent.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  3. Verify that billing is enabled for your Google Cloud project.

  4. Enable the Cloud Monitoring and Observability APIs.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the APIs

  5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  6. Verify that billing is enabled for your Google Cloud project.

  7. Enable the Cloud Monitoring and Observability APIs.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the APIs

    8.

  8. To get the permissions that you need to create a link on an observability dataset, ask your administrator to grant you the Observability Editor (roles/observability.editor) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.

    You might also be able to get the required permissions through custom roles or other predefined roles.

Create dataset

gcloud

Before using any of the command data below, make the following replacements:

  • LINK_ID: The name of the BigQuery dataset.
  • DATASET_ID: The ID of the dataset. Your trace data is stored in a dataset named Spans.
  • BUCKET_ID: The ID of the observability bucket. For example, this ID might be _Trace.
  • LOCATION: The location of the observability buckets.
  • PROJECT_ID: The identifier of the project..

Execute the gcloud beta observability buckets datasets links create command:

Linux, macOS, or Cloud Shell

gcloud beta observability buckets datasets links create \
  projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID/links/LINK_ID \
 --dataset=DATASET_ID\
 --bucket=BUCKET_ID \
 --location=LOCATION \
 --project=PROJECT_ID

Windows (PowerShell)

gcloud beta observability buckets datasets links create `
  projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID/links/LINK_ID `
 --dataset=DATASET_ID`
 --bucket=BUCKET_ID `
 --location=LOCATION `
 --project=PROJECT_ID

Windows (cmd.exe)

gcloud beta observability buckets datasets links create ^
  projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID/links/LINK_ID ^
 --dataset=DATASET_ID^
 --bucket=BUCKET_ID ^
 --location=LOCATION ^
 --project=PROJECT_ID

The create command initiates a long-running operation. The following is an example of a response when the command is successful: 

Create request issued for: [mydataset]
Waiting for operation [projects/my-project/locations/us/operations/operation-1775164903749-64e80c9817833-9ff804b6-c3e9cbe7] to complete...done.
Created link [mydataset].

REST

To create a link to a BigQuery dataset, send a request to the projects.locations.buckets.datasets.links.create endpoint.

You must specify the parent parameter, which has the following form:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID

The fields in the preceding expression have the following meaning:

  • PROJECT_ID: The identifier of the project.
  • LOCATION: The location of the observability bucket.
  • BUCKET_ID: The ID of the observability bucket. For example, this ID might be _Trace.
  • DATASET_ID: The ID of the dataset being queried. For example, this ID might be Spans.

This command requires a query parameter and a request body:

  • The query parameter, linkId, must be specified and set to the name of the BigQuery dataset. For example, linkId="my_link". The BigQuery dataset name must be unique for your Google Cloud project, and must be limited to 100 characters and can include only letters, digits, and underscores.

  • The request body is a Link object. The value of the name field has the following format:

    projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID/links/LINK_ID

    The value you provide for the name field must match the linked BigQuery dataset referenced by the query parameter.

    The LINK_ID field is the name of the BigQuery dataset.

The response is an Operation object. This object contains information about the progress of the method. When the method completes, the Operation object contains status data.

For a complete list of Observability API endpoints, see the Observability API reference documentation.

If you encounter permission errors when creating a linked dataset, then see Troubleshoot permission errors.

What's next