Troubleshoot

You can use this document to resolve common issues when working with Cloud Trace, such as missing span data, query failures in Observability Analytics, and validation errors when creating alerting policies.

Known issues

This section lists known issues:

  • Spans written to your Google Cloud project by using the Telemetry API aren't accessible to the Cloud Trace API. For example, if you try to list these traces, then the command fails with a 404 Not Found error.

Troubleshoot Observability Analytics

This section describes how to resolve failures you might see when using Observability Analytics to query your trace data.

You can't save your alerting policy because of a validation error

You try to save an alerting policy that monitors your trace data and receive an error similar to the following:

The following error occurred when validating your SQL Alert: Error authenticating service account `service-12345@gcp-sa-monitoring-notification.iam.gserviceaccount.com`. BigQuery returned an error.

This error message indicates that the Monitoring Service Account hasn't been granted the required permissions or that it doesn't exist. This account is automatically created by the system when certain user-initiated actions occur. However, if the Cloud Monitoring API is disabled, then the system can't create the service account.

To resolve the failure, do the following:

  1. In the Google Cloud console, go to the APIs & Services page and enable the Cloud Monitoring API:

    Go to APIs & Services

  2. In the Google Cloud console, go to the IAM page:

    Go to IAM

    If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.

  3. On the IAM page, do the following:

    1. Select Include Google-provided role grants.

    2. If the Monitoring Service Account isn't listed, then create a SQL-based alerting policy and try to save the policy.

      When you save the policy, the system creates the Monitoring Service Account. The save action fails because this service account doesn't have the required IAM roles.

    3. Grant the Monitoring Service Account the following roles:

Error message stating a view does not exist

You enter a SQL query in the query pane of the Observability Analytics page, but the SQL parser displays the following error:

projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_ID/datasets/DATASET_ID/views/OBS_VIEW_ID does not exist

The previous error is reported when the view specified in the FROM statement can't be found.

To resolve this error, verify that your view has the proper syntax:

  • Verify that the fully-qualified name of the view follows the syntax required by the Observability Analytics naming scheme. You can find the required syntax for a view by displaying its default query.

  • If the Google Cloud project ID, location, bucket ID, dataset ID, or view ID contain period characters, (.), then verify that the field is wrapped by single backquotes, (`).

    For example, if the ID of your Google Cloud project is example.com:bluebird, then the FROM statement is as follows:

    FROM `example.com:bluebird`.`us`.`_Trace`.`Spans`.`_AllSpans`

Get started with Observability Analytics message is shown

You open the Observability Analytics page and it displays a window with a message similar to the following:

Get started with Observability Analytics

To use Observability Analytics, on the window, click Close.

The previous message is shown when you don't have any log buckets that are upgraded to use Observability Analytics. However, your trace data isn't stored in a log bucket.

Join of multiple views fails

You write a query that joins multiple views, but the query is marked as invalid.

Not all views can be joined.

To join views, the following restrictions apply:

  1. The locations of the views satisfy one of the following:

    • All views have the same location.