Query and analyze traces

You can query and analyze your trace data by using SQL in Observability Analytics to discover performance trends, generate aggregate insights, and display results in charts and tables. You query the default view, which is named _AllSpans, and which includes all trace data stored by your project. You can also save query results to your custom dashboards.

If you want to inspect individual traces or spans, or span attributes, then use the Trace Explorer page. For more information, see Find and explore traces.

About linked BigQuery datasets

You don't need a linked BigQuery dataset to query your trace data, or to query your trace and log data. For those scenarios, you can use the Observability Analytics page. For information about querying log data, see Query and analyze logs with Observability Analytics.

You do need a linked BigQuery dataset when you want to do any of the following:

This document doesn't describe how to create a linked BigQuery dataset or how to configure the Observability Analytics to run queries on reserved slots. For information about these topics, see Create a linked BigQuery dataset.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  3. Verify that billing is enabled for your Google Cloud project.

  4. Enable the Observability API.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the API

  5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  6. Verify that billing is enabled for your Google Cloud project.

  7. Enable the Observability API.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the API

    8.

  8. To get the permissions that you need to load the Observability Analytics page, write, run, and save private queries on your trace data, ask your administrator to grant you the following IAM roles:

    • Observability View Accessor (roles/observability.viewAccessor) on the observability views that you want to query. This role supports IAM conditions, which let you restrict the grant to a specific view. If you don't attach a condition to the role grant, then the principal can access all observability views.
    • Observability Analytics User (roles/observability.analyticsUser) on your project. This role contains the permissions required to save and run private queries, and to run shared queries.

    For more information about granting roles, see Manage access to projects, folders, and organizations.

    You might also be able to get the required permissions through custom roles or other predefined roles.

Display the schema

The schema defines how the data is stored, which includes the fields and their data types. This information is important to you because the schema determines the fields you query and whether you need to cast fields to different data types. For example, to write a query that computes the average latency of HTTP requests, you need to know how to access the latency field and whether it is stored as an integer like 100 or as a string like "100". If the latency data is stored as a string, then the query must cast the value to a numeric value before computing an average.

To identify the schema, do the following:

  1. In the Google Cloud console, go to the Observability Analytics page:

    Go to Observability Analytics

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. In the Views menu, go to the Traces section, and then select _Trace.Spans._AllSpans.

    The Schema pane is updated. Observability Analytics automatically infers the fields of a column when the data type is JSON. To view how often these inferred fields appear in your data, click Options and select View info and description.

    To learn about the schema, see Storage schema for trace data.

    If you don't see a view named _Trace.Spans._AllSpans, then your Google Cloud project doesn't contain a observability bucket named _Trace. For information about how to resolve this failure, see Trace storage initialization fails.

Query trace data

This section describes the approaches that you can use to query your trace data:

  • Load a system-defined query, edit this query, and then run the query.
  • Enter and run a custom query. For example, you might paste in a query you have or write one. Custom queries can include joins, nested queries, and other complex SQL statements. For examples, see Sample SQL queries.
  • Build a query by making menu selections and then run that query. Observability Analytics converts your selections into a SQL query, which you can both view and edit.

Load, edit, and run the system-defined query

  1. In the Google Cloud console, go to the Observability Analytics page:

    Go to Observability Analytics

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. In the Views menu, go to the Traces section, and then select _Trace.Spans._AllSpans.

    If you don't see a view named _Trace.Spans._AllSpans, then your Google Cloud project doesn't contain a observability bucket named _Trace. For information about how to resolve this failure, see Trace storage initialization fails.

  3. Do one of the following:

    • To load a system-defined query that relies on the Query Builder, which lets you define the query with menu selections, make sure that the Query pane displays Query Builder. If a SQL editor is shown, then click  Builder.

    • To load a system-defined query that extracts JSON values, then make sure that Query pane displays the SQL editor. If this pane displays Query Builder, then click  SQL.

  4. In the Schema pane, select Query, and then click Overwrite.

    The Query pane displays a system-defined query. If you selected the Query Builder mode but want to view the SQL query, click  SQL.

  5. Optional: Modify the query.

  6. To run the query, go to the toolbar and select Run Query.

    Observability Analytics presents the query results in a table. However, you can create a chart, and you can also save the table or chart to a custom dashboard. For more information, see Chart SQL query results.

    If the toolbar displays Run in BigQuery, then you need to switch Observability Analytics to use the default query engine. To make this change, in the toolbar of the Query pane, click Settings and then select Analytics (default).

Enter and run a custom query

To enter a SQL query, do the following:

  1. In the Google Cloud console, go to the Observability Analytics page:

    Go to Observability Analytics

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. In the Query pane, click the  SQL.

    • To specify a time range, we recommend that you use the time-range selector. If you add a WHERE clause that specifies the timestamp field, then that value overrides the setting in the time-range selector and that selector is disabled.

    • For examples, see Sample SQL queries.