Monitor Pub/Sub in Cloud Monitoring

You can use the Google Cloud console or the Cloud Monitoring API to monitor Pub/Sub.

This document shows you how to monitor your Pub/Sub usage in the Google Cloud console using Monitoring.

If you want to view metrics from other Google Cloud resources in addition to Pub/Sub metrics, use Monitoring.
Otherwise, you can use the monitoring dashboards provided within Pub/Sub. See Monitor topics and Monitor subscriptions.

For best practices about using metrics in your autoscaling, see Best practices for using Pub/Sub metrics as a scaling signal.

Before you begin

Before you use Monitoring, ensure that you've prepared the following:

A Cloud Billing account
A Pub/Sub project with billing enabled

One way to ensure that you've obtained both is to complete the Quickstart using the Cloud console.

View an existing dashboard

A dashboard lets you view and analyze data from different sources in the same context. Google Cloud provides both predefined and custom dashboards. For example, you can view a predefined Pub/Sub dashboard or create a custom dashboard that displays metric data, alerting policies, and log entries related to Pub/Sub.

To monitor your Pub/Sub project by using Cloud Monitoring, perform the following steps:

In the Google Cloud console, go to the Monitoring page.

Go to Monitoring
Select the name of your project if it is not already selected at the top of the page.
Click Dashboards from the navigation menu.
In the Dashboards overview page, create a new dashboard or select the existing Pub/Sub dashboard.

To search for the existing Pub/Sub dashboard, in the filter for All Dashboards, select the Name property and enter Pub/Sub.

For more information on how to create, edit, and manage a custom dashboard, see Manage custom dashboards.

View a single Pub/Sub metric

To view a single Pub/Sub metric by using the Google Cloud console, perform the following steps:

In the Google Cloud console, go to the Monitoring page.

Go to Monitoring
In the navigation pane, select Metrics explorer.
In the Configuration section, click Select a metric.
In the filter, enter Pub/Sub.
In Active resources, select Pub/Sub Subscription or Pub/Sub Topic.
Drill down to a specific metric and click Apply.

The page for a specific metric opens.

You can learn more about the monitoring dashboard by reading the Cloud Monitoring documentation.

View Pub/Sub metrics and resource types

To see what metrics Pub/Sub reports to Cloud Monitoring, see the Pub/Sub metrics list in the Cloud Monitoring documentation.
To see the details for the pubsub_topic, pubsub_subscription, or pubsub_snapshot monitored resource types, see Monitored resource types in the Cloud Monitoring documentation.

Access the PromQL editor

Metrics Explorer is an interface within Cloud Monitoring designed for exploring and visualizing your metrics data. Within Metrics Explorer, you can use Prometheus Query Language (PromQL) to query and analyze your Pub/Sub metrics.

To access the code editor and query Cloud Monitoring metrics with PromQL in Metrics Explorer, see Use the code editor for PromQL.

For example, you can input a PromQL query to monitor the count of messages sent to a specific subscription over a rolling 1-hour period:

sum(
  increase({
    "__name__"="pubsub.googleapis.com/subscription/sent_message_count",
    "monitored_resource"="pubsub_subscription",
    "project_id"="your-project-id",
    "subscription_id"="your-subscription-id"
  }[1h])
)

Monitor quota usage

For a given project, you can use the IAM & Admin Quotas dashboard to view current quotas and usage.

You can view your historical quota usage by using the following metrics:

These metrics use the consumer_quota monitored resource type. For more quota-related metrics, see the Metrics list.

For example, the following PromQL query creates a chart with the fraction of publisher quota being used in each region:

sum by (quota_metric, location) (
  rate({
    "__name__"="serviceruntime.googleapis.com/quota/rate/net_usage",
    "monitored_resource"="consumer_quota",
    "service"="pubsub.googleapis.com",
    "quota_metric"="pubsub.googleapis.com/regionalpublisher"
  }[${__interval}])
)
/
(max by (quota_metric, location) (
  max_over_time({
    "__name__"="serviceruntime.googleapis.com/quota/limit",
    "monitored_resource"="consumer_quota",
    "service"="pubsub.googleapis.com",
    "quota_metric"="pubsub.googleapis.com/regionalpublisher"
  }[${__interval}])
) / 60 )

If you anticipate your usage exceeding the default quota limits, create alerting policies for all the relevant quotas. These alerts fire when your usage reaches some fraction of the limit. For example, the following PromQL query triggers an alerting policy when any Pub/Sub quota exceeds 80% usage:

sum by (quota_metric, location) (
  increase({
    "__name__"="serviceruntime.googleapis.com/quota/rate/net_usage",
    "monitored_resource"="consumer_quota",
    "service"="pubsub.googleapis.com"
  }[1m])
)
/
max by (quota_metric, location) (
   max_over_time({
    "__name__"="serviceruntime.googleapis.com/quota/limit",
    "monitored_resource"="consumer_quota",
    "service"="pubsub.googleapis.com"
  }[1m])
)
> 0.8

For more customized monitoring and alerting on quota metrics, see Using quota metrics.

See Quotas and limits for more information about quotas.

Maintain a healthy subscription

To maintain a healthy subscription, you can monitor several subscription properties using Pub/Sub-provided metrics. For example, you can monitor the volume of unacknowledged messages, the expiration of message acknowledgment deadlines, and so on. You can also check whether your subscription is healthy enough to achieve a low message delivery latency.

Refer to the next sections to get more details about the specific metrics.

Monitor message backlog

To ensure that your subscribers are keeping up with the flow of messages, create a dashboard. The dashboard can show the following backlog metrics, aggregated by resource, for all your subscriptions:

Unacknowledged messages (subscription/num_unacked_messages_by_region) to see the number of unacknowledged messages.
Oldest unacknowledged message age (subscription/oldest_unacked_message_age_by_region) to see the age of the oldest unacknowledged message in the backlog of the subscription.
Delivery latency health score (subscription/delivery_latency_health_score) to check the overall subscription health in relation to delivery latency. For more information about this metric, see the relevant section of this document.

Create alerting policies that trigger when these values are outside of the acceptable range in the context of your system. For instance, the absolute number of unacknowledged messages is not necessarily meaningful. A backlog of a million messages might be acceptable for a million message-per-second subscription, but unacceptable for a one message-per-second subscription.

Common backlog issues

Symptoms	Problem	Solutions
Both the `oldest_unacked_message_age_by_region` and `num_unacked_messages_by_region` are growing in tandem.	Subscribers not keeping up with message volume	Add more subscriber threads or processes. Add more subscriber machines or containers. Look for signs of bugs in your code that prevent it from successfully acknowledging messages or processing them in a timely fashion. See Monitoring ack deadline expiration.
If there's a steady, small backlog size combined with a steadily growing `oldest_unacked_message_age_by_region`, there may be a few messages that cannot be processed.	Stuck messages	Examine your application logs to understand whether some messages are causing your code to crash. It's unlikely—but possible—that the offending messages are stuck in Pub/Sub rather than in your client. Raise a support case after you're confident your code successfully processes each message. If some messages are causing your code to crash, consider forwarding those messages to a dead-letter topic.
The `oldest_unacked_message_age_by_region` exceeds the subscription message retention duration.	Permanent data loss	Set up an alert that fires before the lapsing of the message retention duration.

Monitor delivery latency health

In Pub/Sub, delivery latency is the time it takes for a published message to be delivered to a subscriber. If your message backlog is increasing, you can use the Delivery latency health score (subscription/delivery_latency_health_score) to check which factors are contributing to an increased latency.

This metric measures the health of a single subscription over a rolling 10-minute window. The metric provides insight into the following criteria, which are necessary for a subscription to achieve consistent low latency: