Instrument for Cloud Trace

You can instrument your applications for Cloud Trace to capture distributed tracing data, examine latency for individual requests, and view the aggregate latency across your services in the Trace console.

This document provides an overview of instrumentation approaches and configuration options. For step-by-step instructions for specific programming languages, see the language-specific setup pages.

When to instrument your application

When trace data for validating performance or troubleshooting issues isn't automatically captured, instrument your application.

Instrument your application to collect specific information that helps you understand its performance and troubleshoot failures. Several open-source instrumentation frameworks collect log, metric, and trace data, and can send that data to any vendor, including Google Cloud. For your agentic applications, some frameworks can collect your prompts and responses or pass context that allows tracing of some remote Google Cloud MCP servers calls.

To instrument your application, we recommend that you use an open-source, vendor-neutral instrumentation framework, such as OpenTelemetry, instead of vendor- and product-specific APIs or client libraries. For information about these frameworks, see Instrumentation and observability and Choose an instrumentation approach.

How to instrument applications

There are several approaches that you can use to instrument your application:

  • Recommended: Use OpenTelemetry, configure your application with an OTLP exporter that sends trace data to a collector, and configure the collector to send trace data to your Google Cloud project by using the Telemetry (OTLP) API. To learn more about our recommendations, see Choose an instrumentation approach.

  • Use OpenTelemetry and configure your application with an OTLP exporter that sends your trace data to your Google Cloud project by using the Telemetry API.

  • If you write applications that run on Compute Engine, then you can use the Ops Agent and the OpenTelemetry Protocol (OTLP) receiver to collect traces and metrics from your application. The Ops Agent can also collect logs, but not by using OTLP. For more information, see Use the Ops Agent and OTLP and Ops Agent overview.

  • Directly invoke either the Telemetry API or the Cloud Trace API.

  • For Spring Boot applications, configure them to forward the trace data they collect to Cloud Trace. For information about this procedure, see Spring Cloud for Google Cloud: Cloud Trace.

  • Use Cloud Trace client libraries or use the Cloud Trace exporter for OpenTelemetry.

Instrumentation samples

The instrumentation samples we provide use OpenTelemetry:

Create custom spans

While OpenTelemetry and client libraries allow you to create custom spans, you might not need to create them manually because these libraries automatically create spans at RPC boundaries.

You can also add information relevant to your application by adding custom annotations and tags to existing spans, or you can create new child spans with their own annotations and tags to trace the application's behavior with finer granularity.

Libraries typically maintain a global trace context that holds information about the current span, including its trace ID and sampling status. Applications can access the current span through the global trace context. Because the context is global, make sure that multi-threaded applications propagate context across threads to maintain accurate trace data.

Force trace sampling

You can't force spans to be sampled because each component in the request path makes an independent sampling decision. However, you can influence downstream components by setting the sampled flag in the trace header to true. This setting is a hint to child components to sample the request. For more information about trace headers, see Protocols for context propagation.

  • Your applications: You configure how the instrumentation logic respects the sampled flag. For example, when using OpenTelemetry, you can use the ParentBased sampler to ensure that the parent's sampling flag is respected.

  • Google Cloud services: Each service determines its own tracing support. In general, services accept the parent sampling flag as a hint while enforcing their own sampling rate limits.

Correlate metrics and traces with exemplars

You can correlate metric data with traces by using exemplars. An exemplar is a representative sample request or span associated with a metric measurement. For example, an exemplar can contain a link to a trace, which lets you correlate your metric and trace data. For an OpenTelemetry-based example, see Correlate metrics and traces by using exemplars.

You might see system-generated exemplars on dashboard charts that display SQL query results for trace data. These exemplars link specific query results directly to traces. For more information, see Generate and display trace exemplars.

Configure your project and platform

This section describes the required APIs and Identity and Access Management (IAM) roles, and explains how to configure authentication credentials for your platform..

Enable APIs

By default, Google Cloud projects have the Cloud Trace API and Telemetry API enabled, and you don't need to take any action. However, security constraints defined by your organization might have disabled one or both of these APIs. For troubleshooting information, see Develop applications in a constrained Google Cloud environment.

Enable the Telemetry and Cloud Trace APIs.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

Grant IAM roles

The required IAM roles depend on whether you are viewing trace data in the Google Cloud console or writing trace data to your project:

  • To get the permissions that you need to view trace data by using the Google Cloud console, ask your administrator to grant you the Cloud Trace User (roles/cloudtrace.user) IAM role on your project.

  • To get the permissions that you need to write trace data by using the Cloud Trace API, ask your administrator to grant you the Cloud Trace Agent (roles/cloudtrace.agent) IAM role on your project.

  • To get the permissions that you need to write trace data by using the Telemetry API, ask your administrator to grant you the Cloud Telemetry Writer role (roles/telemetry.writer) IAM role on your project.

Authenticate

This section describes how to authenticate when your applications run on Google Cloud and when they run elsewhere.

Run on Google Cloud

When your application runs on Google Cloud, you generally don't need to provide authentication credentials. However, some language client libraries require the project ID even when hosted on Google Cloud.

Verify that your Google Cloud platform has the Cloud Trace API access scope enabled. For the following configurations, the default access-scope settings include the Cloud Trace API access scope:

If you use custom access scopes, then you must ensure that Cloud Trace API access scope is enabled. For example, if you use the Google Cloud CLI to create a GKE cluster and if you specify the --scopes flag, then ensure the scope includes trace.append. The following command illustrates setting the --scopes flag:

gcloud container clusters create example-cluster-name --scopes=https://www.googleapis.com/auth/trace.append

Run locally and elsewhere

If your application runs outside Google Cloud, then you must provide authentication credentials to the client library. The service account must be granted the Cloud Trace Agent role (roles/cloudtrace.agent). For information about roles, see Control access with IAM.

Google Cloud client libraries use Application Default Credentials (ADC) to find your application's credentials. You can provide these credentials in one of three ways:

  • Run gcloud auth application-default login

  • Place the service account key file in a default path for your operating system. The following lists the default paths for Windows and Linux:

    • Windows: %APPDATA%/gcloud/application_default_credentials.json

    • Linux: $HOME/.config/gcloud/application_default_credentials.json

  • Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path to your service account:

    Linux/macOS

        export GOOGLE_APPLICATION_CREDENTIALS=path-to-your-service-accounts-private-key

    Windows

        set GOOGLE_APPLICATION_CREDENTIALS=path-to-your-service-accounts-private-key

    PowerShell:

        $env:GOOGLE_APPLICATION_CREDENTIALS="path-to-your-service-accounts-private-key"

What's next