Create a search data store

Console

To use the Google Cloud console to make a data store and index websites, follow these steps:

1. In the Google Cloud console, go to the AI Applications page.

   AI Applications

2. In the navigation menu, click Data Stores.

3. Click Create data store.

4. On the Source page, select Website Content.

5. Choose whether to turn on Advanced website indexing for this data store. If you turn advanced website indexing on now, you can't turn it off later.

   Advanced website indexing provides additional features such as search summarization, search with follow-ups, and extractive answers. Advanced website indexing incurs additional cost, and requires that you verify domain ownership for any website that you index. For more information, see Advanced website indexing and Pricing.

6. If you select Advanced website indexing, select one of the Indexing Options:

   This choice can't be changed later.

   Option	Description
   Automatic URL Discovery & Continuous Updates	This indexes the URLs that you specify if those pages are known to Google. A very new page won't be known to Google because the Google web crawler won't have crawled it. After the initial indexing, this discovers new pages and refreshes updated pages on a best-effort basis. (This can result in relatively stale pages.) For more information, see Refresh web pages using automatic and manual refresh. This creates a data store where Automatic Refresh is Enabled and Sitemap Mode is set to Discovery.
   Specific URLs + Google's Known URLs	This indexes the pages in the sitemaps and the pages that match the URL patterns if the pages are known to Google. Because this option doesn't enable automatic refresh, you refresh the index by running manual refresh and by updating the sitemap. For more information, see Manual refresh and Index and refresh web pages using sitemaps. This creates a data store where Automatic Refresh is Disabled and Sitemap Mode is set to Discovery.
   Specific URLs from Sitemap and Manually Specified URLs	This gives you full control over what gets indexed. Indexing is based on the sitemap. When you need to update the index, see Index and refresh web pages using sitemaps. This creates a data store where Automatic Refresh is Disabled and Sitemap Mode is set to Exact.

7. In the Sites to include field, enter the URL patterns matching the websites that you want to include in your data store. Include one URL pattern per line, without comma separators. For example, example.com/docs/*

8. In the Sites to exclude field, enter URL patterns that you want to exclude from your data store.

   Excluded sites take priority over included sites. So, if you were to include example.com/docs/* but exclude example.com, then no websites would be indexed.

   For more information, see Website data in Prepare data for ingesting.

9. Click Continue.

10. Select a location for your data store.

    • When you create a basic website search data store, this is always set to global (Global).
    • When you create a data store with advanced website indexing, you can select a location. Because the websites that are indexed must be public, Google strongly recommends that you select global (Global) as your location. This ensures maximum availability of all search and answering services and eliminates the limitations of regional data stores.

11. Enter a name for your data store.

12. Click Create. Agent Search creates your data store and displays your data stores on the Data Stores page.

13. To view information about your data store, click the name of your data store in the Name column. Your data store page appears.

    • If you turned on Advanced website indexing, a warning appears prompting you to verify the domains in your data store.
    • If you have a quota shortfall (the number of pages in the websites that you specified exceeds the "Number of documents per project" quota for your project), an additional warning appears prompting you to upgrade your quota.

14. To verify the domains for the URL patterns in your data store, follow the instructions on the Verify website domains page.

15. To upgrade your quota, follow these steps:

    1. Click Upgrade quota. The IAM and Admin page of the Google Cloud console appears.
    2. Follow the instructions at Request a quota adjustment in the Google Cloud documentation. The quota to increase is Number of documents in the Discovery Engine API service.
    3. After submitting your request for a higher quota limit, go back to the AI Applications page and click Data Stores in the navigation menu.
    4. Click the name of your data store in the Name column. The Status column indicates that indexing is in progress for the websites that had surpassed the quota. When the Status column for a URL shows Indexed, advanced website indexing features are available for that URL or URL pattern.

    For more information, see Quota for web page indexing in the "Quotas and limits" page.

Python

For more information, see the Agent Search Python API reference documentation.

To authenticate to Agent Search, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

Create a data store

from google.api_core.client_options import ClientOptions
from google.cloud import discoveryengine

# TODO(developer): Uncomment these variables before running the sample.
# project_id = "YOUR_PROJECT_ID"
# location = "YOUR_LOCATION" # Values: "global"
# data_store_id = "YOUR_DATA_STORE_ID"


def create_data_store_sample(
    project_id: str,
    location: str,
    data_store_id: str,
) -> str:
    #  For more information, refer to:
    # https://cloud.google.com/generative-ai-app-builder/docs/locations#specify_a_multi-region_for_your_data_store
    client_options