Refresh web pages using automatic and manual refresh

If your data store uses basic website search, the freshness of your store's index mirrors the freshness that's available in Google Search.

If advanced website indexing is enabled in your data store, the web pages in your data store are refreshed in the following ways:

  • Automatic refresh
  • Manual refresh
  • Sitemap-based refresh

This page describes automatic and manual refresh. To understand and implement sitemap-based refresh, see Index and refresh according to sitemap.

Before you begin

If you use the robots.txt file in your website, update it. For more information, see how to prepare your website's robots.txt file.

Automatic refresh

Agent Search performs automatic refresh as follows:

  • After you create a data store, it generates an initial index for the included pages.
  • After the initial indexing, it indexes any newly discovered pages and recrawls existing pages on a best-effort basis.
  • It regularly refreshes data stores that encounter a query rate of 50 queries/30 days.

Manual refresh

If you want to refresh specific web pages in a data store with Advanced website indexing turned on, you can call the recrawlUris method. You use the uris field to specify each web page that you want to crawl. The recrawlUris method is a long-running operation that runs until your specified web pages are crawled or until it times out after 24 hours, whichever comes first. If the recrawlUris method times out you can call the method again, specifying the web pages that remain to be crawled. You can poll the operations.get method to monitor the status of your recrawl operation.

Limits on recrawling

There are limits to how often you can crawl web pages and how many web pages that you can crawl at a time:

  • Calls per day. The maximum number of calls to the recrawlUris method allowed is 20 per day, per project.
  • Web pages per call. The maximum number of uris values that you can specify with a call to the recrawlUris method is 10,000.

Recrawl the web pages in your data store

You can manually crawl specific web pages in a data store that has Advanced website indexing turned on.

REST

To use the command line to crawl specific web pages in your data store, follow these steps:

  1. Find your data store ID. If you already have your data store ID, skip to the next step.

    1. In the Google Cloud console, go to the AI Applications page and in the navigation menu, click Data Stores.

      Go to the Data Stores page

    2. Click the name of your data store.

    3. On the Data page for your data store, get the data store ID.

  2. Call the recrawlUris method, using the uris field to specify each web page that you want to crawl. Each uri represents a single page even if it contains asterisks (*). Wildcard patterns are not supported.