This page describes how to create a data store and ingest data for custom search apps in Agent Search. Go to the section for the source you plan to use:
- Create a data store using website content
- Import from BigQuery
- Import from Cloud Storage
- Sync from Google Drive
- Sync from Gmail
- Sync from Google Sites
- Sync from Google Calendar
- Sync from Google Groups
- Import from Cloud SQL
- Import from Spanner
- Import from Firestore
- Import from Bigtable
- Import from AlloyDB for PostgreSQL
- Upload structured JSON data with the API
- Create a data store using Terraform
To sync data from a third-party data source instead, see Connect a third-party data source.
For troubleshooting information, see Troubleshoot data ingestion.
To create data stores and connect data for Gemini Enterprise apps, see Introduction to connectors and data stores.
Create a data store using website content
Use the following procedure to create a data store and index websites.
To use a website data store after creating it, you must attach it to an app that has Enterprise features turned on. You can turn on Enterprise Edition for an app when you create it. This incurs additional costs. See Create a search app and About advanced features.
Before you begin
Complete the items listed in Website data.
Procedure
Console
To use the Google Cloud console to make a data store and index websites, follow these steps:
In the Google Cloud console, go to the AI Applications page.
In the navigation menu, click Data Stores.
Click Create data store.
On the Source page, select Website Content.
Choose whether to turn on Advanced website indexing for this data store. If you turn advanced website indexing on now, you can't turn it off later.
Advanced website indexing provides additional features such as search summarization, search with follow-ups, and extractive answers. Advanced website indexing incurs additional cost, and requires that you verify domain ownership for any website that you index. For more information, see Advanced website indexing and Pricing.
If you select Advanced website indexing, select one of the Indexing Options:
This choice can't be changed later.
Option Description Automatic URL Discovery & Continuous Updates This indexes the URLs that you specify if those pages are known to Google. A very new page won't be known to Google because the Google web crawler won't have crawled it.
After the initial indexing, this discovers new pages and refreshes updated pages on a best-effort basis. (This can result in relatively stale pages.) For more information, see Refresh web pages using automatic and manual refresh.
This creates a data store where Automatic Refresh is Enabled and Sitemap Mode is set to Discovery.