Working with list operations

Some services return potentially large lists of items, such as rows or resource descriptions. To keep CPU and memory usage under control, services return these resources in pages: smaller subsets of the items with a continuation token to request the next subset.

Iterating over items by page can be tedious. The client libraries provide adapters to convert the pages into asynchronous iterators. This guide shows you how to work with these adapters.

Prerequisites

This guide uses the Secret Manager service to demonstrate list operations. The concepts included in this guide also apply to other services.

To follow this guide, you'll need to to have enabled the Secret Manager service, logged in, and ensured that your account has the necessary permissions. For guidance on how to fulfill these prerequisites, follow the service quickstart.

For complete setup instructions for the Rust libraries, see Setting up your development environment.

Dependencies

Add the Secret Manager library to your Cargo.toml file:

cargo add google-cloud-secretmanager-v1

Iterating list methods

To help iterate the items in a list method, the APIs return an implementation of the ItemPaginator trait. Introduce the trait into scope using a use declaration:

use google_cloud_gax::paginator::ItemPaginator as _;

To iterate over the items, use the by_item method.

let mut list = client
    .list_secrets()
    .set_parent(format!("projects/{project_id}"))
    .by_item();
while let Some(secret) = list.next().await {
    let secret = secret?;
    println!("  secret={}", secret.name)
}

In rare cases, pages might contain extra information that you need access to, or you may need to checkpoint your progress across processes. If needed, you can iterate over full pages instead of individual items.

1. Introduce Paginator into scope using a use declaration:

   use google_cloud_gax::paginator::Paginator as _;

2. Iterate over the pages using by_page:

   let mut list = client
       .list_secrets()
       .set_parent(format!("projects/{project_id}"))
       .by_page();
   while let Some(page) = list.next().await {
       let page = page?;
       println!("  next_page_token={}", page.next_page_token);
       page.secrets
           .into_iter()
           .for_each(|secret| println!("    secret={}", secret.name));
   }

Working with futures::Stream

You may want to use these APIs in the larger Rust ecosystem of asynchronous streams, such as tokio::Stream. This can be achieved as follows:

1. Enable the unstable-streams feature in the google_cloud_gax crate. The name of this feature serves as notice that these APIs are unstable. You should only use them if you are prepared to deal with any breaks that result from incompatible changes to the futures::Stream trait.

   cargo add google-cloud-gax --features unstable-stream
   

2. The following examples also use the futures::stream::StreamExt trait, which you enable by adding the futures crate.

   cargo add futures
   

3. Add the required use declarations:

   use futures::stream::StreamExt;
   use google_cloud_gax::paginator::ItemPaginator as _;

4. Use the into_stream function to convert ItemPaginator into a futures::Stream.

   let list = client
       .list_secrets()
       .set_parent(format!("projects/{project_id}"))
       .by_item()
       .into_stream();
   list.map(|secret| -> gax::Result<()> {
       println!("  secret={}", secret?.name);
       Ok(())
   })
   .fold(Ok(()), async |acc, result| -> gax::Result<()> {
       acc.and(result)
   })
   .await?;

Similarly, you can use the into_stream function to convert Paginator into a futures::Stream.

let list = client