Find and explore traces

You can view an aggregate representation of your trace data, filter your trace data, and explore individual traces by using the Trace Explorer page. By default, the Trace Explorer queries the data stored in your Google Cloud project. However, you can configure the page to query trace data stored in multiple projects.

The following screenshot illustrates how trace data is shown:

To perform a custom aggregate analysis of your spans using SQL, use the Observability Analytics page or BigQuery. When you use the Observability Analytics page, you can view query results as a table or as a chart. For information about using SQL to query your trace data, see Query and analyze traces.

About the Trace Explorer page

To help you identify trends and patterns in your trace data, the latency data is aggregated and displayed in charts. The heatmap uses color to represent the number of spans in a cell. A cell with many spans has a darker color than one with few spans. Each cell's tooltip displays latency and other information. You can also view latency percentiles and span rate information.

When you are investigating an issue, you might want to view a specific trace or only spans with certain properties:

• To restrict which spans are shown, apply filters. For example, you can filter by service, application, error state, and other properties.

• To display a specific trace, use the toolbar's pageview Search for trace option.

To identify outliers and view details of individual spans, use the tabular section:

• You can sort the rows of the Spans and Grouped tabs by the duration header.

• To understand how many spans different services are generating, use the Grouped tab. Each row corresponds to a service and span name, and for each entry, latency statistics are shown.

To explore details of a span or grouping, select the entry from a row in the tabular section. A flyout opens that displays the span in the context of its trace. From this view, you can explore the span, its attributes, and view events and log data. You can also search the spans and attributes in the trace for keywords.

The trace data that the Trace Explorer page displays depends on the following:

• The Scope setting, which determines which views the system searches for trace data. When the page opens, the Scope element is set to the default trace scope and the system queries the views listed in that scope for trace data.

  On a new project, the default trace scope lists the _AllSpans view for the project. That view provides read access to all trace data that the project stores.

  You can create trace scopes that let you query data stored in multiple projects. For more information, see View traces across projects.

• Your Identity and Access Management (IAM) permissions on the searched view. If you don't have permission to view trace data for a project, then the Google Cloud console displays a warning message and the data from that project isn't displayed.

• The time-range setting.

• The filters you apply.

You can save your trace data on a custom dashboard. For more information, see Display trace data on a custom dashboard.

The remainder of this page provides more information about how to find and explore your trace data.

Before you begin

To get the permissions that you need to view trace data by using the Google Cloud console and to select a trace scope, ask your administrator to grant you the Cloud Trace User (roles/cloudtrace.user) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the permissions required to view trace data by using the Google Cloud console and to select a trace scope. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to view trace data by using the Google Cloud console and to select a trace scope:

• To select a trace scope: cloudtrace.traceScopes.[get, list]
• To read the default trace scope: observability.scopes.get

You might also be able to get these permissions with custom roles or other predefined roles.

For more information about roles, see Control access with Identity and Access Management.

View aggregated trace data

To view the aggregated information about your trace data, do the following:

1. In the Google Cloud console, go to the Trace explorer page:

   Go to Trace explorer

   You can also find this page by using the search bar.

   It might take several minutes after the first trace data is written to a Google Cloud project before that data is available to view. If you don't see any trace data displayed after waiting a few minutes, then your project might not have any data to display or there might be a configuration issue. For information about how to resolve these issues, see Troubleshoot: No data in the Trace interface.

2. Configure the Trace Explorer page:

   1. In the toolbar of the Google Cloud console, select your Google Cloud project. For App Hub configurations, select the App Hub host project or management project.

   2. Optional: Configure which views the system searches for trace data by using the Scope element:

      • To show the trace data that your project stores, set the first menu of the Scope element to Project or to _Default. These two settings are equivalent.

      • To show the trace data that multiple projects store, expand the first menu of the Scope element, select Trace scope, and then select the trace scope that lists the _AllSpans view for those projects. After you make your selection, the Scope menu displays a trace scope icon, , and the name of the selected trace scope.

      The data that the system displays depends on your IAM roles on the searched views. For example, if you don't have the permissions necessary to see data in a view, then the system doesn't display that data.

      For more information, see Create and manage trace scopes.