Exactly-once delivery

This page explains how to receive and acknowledge messages using Pub/Sub's exactly-once feature, which makes it possible for you to track and prevent duplicate processing of messages. When the feature is enabled, Pub/Sub provides the following semantics:

• Subscribers can determine if message acknowledgments were successful.

• No redelivery occurs after the message is successfully acknowledged.

• No redelivery occurs while a message is outstanding. A message is considered outstanding until the acknowledgment deadline expires or the message is acknowledged.

• In case of multiple valid deliveries, due to acknowledgment deadline expiration or client-initiated negative acknowledgment, only the latest acknowledgment ID can be used to acknowledge the message. Any requests with a previous acknowledgment ID fail.

With exactly-once enabled, subscribers can ensure messages are processed one time by following these guidelines:

• Acknowledge messages within the acknowledgment deadline.

• Maintain information about the progress of processing a message until it is successfully acknowledged.

• Use the information about the progress of processing a message to prevent duplicate work when an acknowledgment fails.

Only the pull subscription type supports exactly-once delivery, including subscribers that use the StreamingPull API. Push and export subscriptions don't support exactly-once delivery.

Pub/Sub supports exactly-once delivery, within a cloud region, based on a Pub/Sub-defined unique message ID.

Recommended client library versions

• For best performance, use the latest version of the client library, Python v2.13.6 or higher, Java v1.139.0 or higher, PHP v1.39.0 or higher, C# v3.2.0 or higher, C++ v2.1.0, Go v1.25.1 or higher, Node v3.2.0 or higher and Ruby v2.12.1 or higher.

Redelivery versus duplicate

It is important to understand the difference between expected and unexpected redeliveries.

• A redelivery can happen either because of client-initiated negative acknowledgment of a message or when the client doesn't extend the acknowledgment deadline of the message before the acknowledgment deadline expires. Redeliveries are considered valid and system working as intended.

  To troubleshoot redeliveries, see Dealing with duplicates.

• A duplicate is when a message is re-sent after a successful acknowledgment or before acknowledgment deadline expiration.

• A redelivered message retains the same message ID between redelivery attempts.

Subscriptions with exactly-once delivery enabled don't receive duplicate deliveries.

Exactly-once delivery support in client libraries

• Supported client libraries have an interface for acknowledgment with response (example: Go). You can use this interface to check if the acknowledgment request succeeded. If the acknowledgment request succeeds, the clients are guaranteed to not receive a re-delivery. If the acknowledgment request fails, the clients can expect a re-delivery.

• Clients can also use the supported client libraries without the acknowledgment interface. However, in such cases, the acknowledgment failures can lead to silent re-deliveries of messages.

• Supported client libraries have interfaces for setting the minimum lease extension time (example: Go). You must set the value for the minimum lease extension to a high number to avoid any network-related acknowledgment expirations. The maximum value is set at 600 seconds.

• If you are using the Java client library and you initialize your subscriber with a custom gRPC Channel using the setChannelProvider() method, it is recommended that you also set maxInboundMetadataSize to at least 1MB when building your TransportChannelProvider. For this configuration, you can use the InstantiatingGrpcChannelProvider.Builder.setMaxInboundMetadataSize() or the ManagedChannelBuilder.maxInboundMetadataSize() method.

The default values and range for the variables related to exactly-once delivery and the names of the variables might differ across client libraries. For example, in the Java client library, the following variables control exactly-once delivery.

In the case of exactly-once delivery, the modifyAckDeadline or acknowledgment request to Pub/Sub fails when the acknowledgment ID is already expired. In such cases, the service considers the expired acknowledgment ID as invalid, since a newer delivery might already be in-flight. This is by design for exactly-once delivery. You then see acknowledgment and ModifyAckDeadline requests return an INVALID_ARGUMENT response. When exactly-once delivery is disabled, these requests return OK in cases of expired acknowledgment IDs.

To ensure that acknowledgment and ModifyAckDeadline requests have valid acknowledgment IDs, consider setting the value for minDurationPerAckExtension to a high number.

Regional considerations

The exactly-once delivery guarantee only applies when subscribers connect to the service in the same region. If your subscriber application is spread across multiple regions, it can lead to duplicate message delivery, even when exactly-once delivery is enabled. Publishers can send messages to any region and the exactly once guarantee is still maintained.

When you run your application within Google Cloud, by default it connects to the Pub/Sub endpoint in the same region. Therefore, running your application in a single region within Google Cloud generally ensures you are interacting with a single region.

When you are running your subscriber application outside of Google Cloud or in multiple regions, you can guarantee you are connecting to a single region by using a locational endpoint when configuring your Pub/Sub client. All location endpoints for Pub/Sub point to single regions. To learn more about locational endpoints, see Pub/Sub endpoints. For a list of all locational endpoints for Pub/Sub, see List of locational endpoints.

Create subscriptions with exactly-once delivery

You can create a subscription with exactly-once delivery using the Google Cloud console, the Google Cloud CLI, client library, or the Pub/Sub API.

Pull subscription

gcloud pubsub subscriptions create SUBSCRIPTION_ID \
  --topic=TOPIC_ID \
  --enable-exactly-once-delivery

PUT https://pubsub.googleapis.com/v1/projects/PROJECT_ID/subscriptions/SUBSCRIPTION_ID
Authorization: Bearer $(gcloud auth print-access-token)

{
  "topic": "projects/PROJECT_ID/topics/TOPIC_ID",
  "enableExactlyOnceDelivery": true,
}

namespace pubsub = ::google::cloud::pubsub;
namespace pubsub_admin = ::google::cloud::pubsub_admin;
[](pubsub_admin::SubscriptionAdminClient client,
   std::string const& project_id, std::string const& topic_id,
   std::string const& subscription_id) {
  google::pubsub::v1::Subscription request;
  request.set_name(
      pubsub::Subscription(project_id, subscription_id).FullName());
  request.set_topic(pubsub::Topic(project_id, topic_id).FullName());
  request.set_enable_exactly_once_delivery(true);
  auto sub = client.CreateSubscription(request);
  if (sub.status().code() == google::cloud::StatusCode::kAlreadyExists) {
    std::cout << "The subscription already exists\n";
    return;
  }
  if (!sub) throw std::move(sub).status();

  std::cout << "The subscription was successfully created: "
            << sub->DebugString() << "\n";
}