Create dataset

A labeled dataset of documents is required to train, up-train, or evaluate a processor version.

This page describes how to create a dataset, import documents, and define a schema. To label the imported documents, see Label documents.

This page assumes you have already created a processor that supports training, up-training, or evaluation. If your processor is supported, you see the Train tab in the Google Cloud console.

Dataset storage options

You can choose between two options to save your dataset:

  • Google-managed
  • Custom location Cloud Storage

Unless you have special requirements (for example to keep documents in a set of CMEK-enabled folders) we recommend the simpler Google-managed storage option. Once created the dataset storage option cannot be changed for the processor.

The folder or subfolder for a custom Cloud Storage location must start empty and be treated as strictly read-only. Any manual changes to its contents might make the dataset unusable, risking its loss. The Google-managed storage option does not have this risk.

Follow these steps to provision your storage location.

  1. Display advanced options while creating a new processor.

    create-dataset-1

  2. Keep the default radio group option to Google-managed storage.

    create-dataset-2

  3. Select Create.

    create-dataset-3

  4. Confirm dataset is created successfully and dataset location is Google-managed location.

    create-dataset-4

Custom storage option

  1. Set the advanced options on or off.

    create-dataset-1

  2. Select I'll specify my own storage location.

    create-dataset-5

  3. Choose a Cloud Storage folder from the input component.

    create-dataset-6

  4. Select Create.

    create-dataset-7

Dataset API operations

This sample shows you how to use the processors.updateDataset method to create a dataset. A dataset resource is a singleton resource in a processor, which means that there is no create resource RPC. Instead, you can use the updateDataset RPC to set the preferences. Document AI provides an option to store the dataset documents in a Cloud Storage bucket you provide or to have them automatically managed by Google.

Before using any of the request data, make the following replacements:

LOCATION: Your processor location
PROJECT_ID: Your Google Cloud project ID
PROCESSOR_ID The ID of your custom processor
GCS_URI: Your Cloud Storage URI where dataset documents are stored

Provided bucket

Follow next steps to create a dataset request with a Cloud Storage bucket you provide.

HTTP method

PATCH https://LOCATION-documentai.googleapis.com/v1beta3/projects/PROJECT_ID/locations/LOCATION/processors/PROCESSOR_ID/dataset

Request JSON:

  {
      "name":"projects/PROJECT_ID/locations/LOCATION/processors/PROCESSOR_ID/dataset"
      "gcs_managed_config" {
          "gcs_prefix" {
              "gcs_uri_prefix": "GCS_URI"
          }
      }
      "spanner_indexing_config" {}
  }

Google managed

In case you want to create the dataset which is Google managed, update the following information:

HTTP method

PATCH https://LOCATION-documentai.googleapis.com/v1beta3/projects/PROJECT_ID/locations/LOCATION/processors/PROCESSOR_ID/dataset

Request JSON:

  {
      "name":"projects/PROJECT_ID/locations/LOCATION/processors/PROCESSOR_ID/dataset"
      "unmanaged_dataset_config": {}
      "spanner_indexing_config": {}
  }

To send your request, you can use Curl: