You can create a linked BigQuery dataset on your observability dataset, which lets you use BigQuery services to query your trace data. You can then join trace data with business datasets and analyze spans by using BigQuery Studio or the BigQuery API.
You don't need a linked BigQuery dataset to analyze your trace data with SQL if you use Observability Analytics. You can also join your trace and log data when you use Observability Analytics. For more information, see Query and analyze traces with SQL and Query and analyze logs with Observability Analytics.
When you need a linked dataset
You need a linked BigQuery dataset when you want to do any of the following:
- Join trace data with other BigQuery datasets.
- Query trace data from another service like BigQuery Studio or Data Studio.
Run queries from the Observability Analytics page on the BigQuery engine.
The Observability Analytics page automatically runs queries on the default query engine, which uses free slots. However, you can configure this page to run queries on the BigQuery engine.
Create a linked BigQuery dataset
This section describes how to create a linked BigQuery dataset on an observability dataset, which lets you use BigQuery services to query your trace data. Each observability dataset supports one linked BigQuery dataset.
When you create a linked dataset, you initiate a long-running operation. Several actions occur as part of that operation:
- Audit logs record the request to create a link and the completion of the operation.
When the Monitoring Service Account doesn't exist, Google Cloud Observability creates it.
The Cloud Monitoring API must be enabled for this service account to be created.
Google Cloud Observability requires a service account before it can create a linked dataset. When necessary, a service account is created:
Cloud Observability Service Account: Required for a link on an observability dataset.
Cloud Logging Service Account: Required for a link on a log bucket.
An audit log records the request from the service agent manager to grant the Monitoring Service Account the IAM role of Monitoring Service Agent.
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
-
Create a project: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles.
-
Verify that billing is enabled for your Google Cloud project.
Enable the Cloud Monitoring and Observability APIs.
Roles required to enable APIs
To enable APIs, you need the
serviceusage.services.enablepermission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
-
Create a project: To create a project, you need the Project Creator role
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles.
-
Verify that billing is enabled for your Google Cloud project.
Enable the Cloud Monitoring and Observability APIs.
Roles required to enable APIs
To enable APIs, you need the
serviceusage.services.enablepermission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.-
To get the permissions that you need to create a link on an observability dataset, ask your administrator to grant you the Observability Editor (
roles/observability.editor) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.You might also be able to get the required permissions through custom roles or other predefined roles.
Create dataset
gcloud
Before using any of the command data below, make the following replacements:
- LINK_ID: The name of the BigQuery dataset.
- DATASET_ID: The ID of the dataset. Your trace data is stored in a dataset named
Spans. - BUCKET_ID: The ID of the observability bucket. For example, this ID might be
_Trace. - LOCATION: The location of the observability buckets.
- PROJECT_ID: The identifier of the project..
Execute the
gcloud beta observability buckets datasets links create
command:
Linux, macOS, or Cloud Shell
gcloud beta observability buckets datasets links create \ projects/