Copy processor versions and datasets across projects

This page contains steps to copy Document AI trained processor versions from one project to another along with dataset schema and samples from the source to destination processor. These steps automate the process of importing the processor version, deploying it, and setting it as the default version in the destination project.

Before you begin

  • Get a Google Cloud Project ID.
  • Have Document AI Processor ID.
  • Have Cloud Storage.
  • Use Python: Jupyter notebook (Vertex AI).
  • Need permissions to give access the service account in the source and destination projects.

Step-by-step procedure

The procedure is described in the following steps.

Step 1: Identify the service account associated with Vertex AI Notebook

!gcloud config list account

Output:

[core]
account = example@automl-project.iam.gserviceaccount.com

Your active configuration is: [default]

Step 2: Grant required permissions to the service account

In the Google Cloud project that is the intended destination for migration, add the service account that was acquired in the previous step as a principal and assign the two following roles:

  • Document AI Administrator
  • Storage Admin

See Granting roles to service accounts and Customer-managed encryption keys (CMEK) for more information.

processor-version-migrate-1

For the migration to work, the service account used for running this notebook needs to have:

  • Roles in both source and destination projects to create the dataset bucket, or create it if it does not exist, as well as read and write permissions to all objects.
  • Document AI Editor role in the source project as described in Import a processor version.

Download a JSON key for the service account, so that you can authenticate and authorize as Service Account. For more on this, see Service account keys.

Next:

  1. Go to the service account.
  2. Select the service account intended to perform this task.
  3. Go to the Keys tab, and click Add Key, then choose Create new key.
  4. Select the key type (preferably JSON).

  5. Click Create and download to specific path. processor-version-migrate-2

  6. Update path in service_account_key variable in the following snippet.

service_account_key='path_to_sa_key.json'

from google.oauth2 import service_account
from google.cloud import storage

# Authenticate the service account
credentials = service_account.Credentials.from_service_account_file(
    service_account_key
)

# pass this credentials variable to all client initializations
# storage_client = storage.Client(credentials=credentials)
# docai_client = documentai.DocumentProcessorServiceClient(credentials=credentials)

Step 3: Import libraries

import time
from pathlib import Path
from typing import Optional, Tuple
from google.cloud.documentai_v1beta3.services.document_service import pagers
from google.api_core.client_options import ClientOptions
from google.api_core.operation import Operation
from google.cloud import documentai_v1beta3 as documentai
from google.cloud import storage
from tqdm import tqdm

Step 4: Input details

  • source_project_id: Provide source project ID.
  • source_location: Provide Source Processor Location (us or eu).
  • source_processor_id: Provide Google Cloud Document AI Processor ID.
  • source_processor_version_to_import: Provide Google Cloud Document AI Processor Version ID for the trained version.
  • migrate_dataset: Provide this value as either True or False, if you want to migrate dataset from source processor to destination processor then provide True, else False. The default value is False.
  • source_exported_gcs_path: Provide Cloud Storage path to store JSON files.
  • destination_project_id: Provide destination project ID.
  • destination_processor_id: Provide Google Cloud Document AI Processor ID, either "" or processor_id from destination project.
source_project_id = "source-project-id"
source_location = "processor-location"
source_processor_id = "source-processor-id"
source_processor_version_to_import = "source-processor-version-id"
migrate_dataset = False  # Either True or False
source_exported_gcs_path = (
    "gs://bucket/path/to/export_dataset/"
)
destination_project_id = "< destination-project-id >"
# Give an empty string if you wish to create a new processor
destination_processor_id = ""

Step 5: Run the code

import time
from pathlib import Path
from typing import Optional, Tuple
from google.cloud.documentai_v1beta3.services.document_service import pagers
from google.api_core.client_options import ClientOptions
from google.api_core.operation import Operation
from google.cloud import documentai_v1beta3 as documentai
from google.cloud import storage
from tqdm import tqdm

source_project_id = "source-project-id"
source_location = "processor-location"
source_processor_id = "source-processor-id"
source_processor_version_to_import = "source-processor-version-id"
migrate_dataset = False # Either True or False
source_exported_gcs_path = (
    "gs://bucket/path/to/export_dataset/"
)
destination_project_id = "< destination-project-id >"
# Give empty string if you wish to create a new processor
destination_processor_id = ""

exported_bucket_name = source_exported_gcs_path.split("/")[2]
exported_bucket_path_prefix = "/".join(source_exported_gcs_path.split("/")[3:])
destination_location = source_location

def sample_get_processor(project_id: str, processor_id: str, location: str)->Tuple[str, str]:
    """
    This function returns Processor Display Name and Type of Processor from source project

    Args:
        project_id (str): Project ID
        processor_id (str): Document AI Processor ID
        location (str): Processor Location

    Returns:
        Tuple[str, str]: Returns Processor Display name and type
    """
    client = documentai.DocumentProcessorServiceClient()
    print(
        f"Fetching processor({processor_id}) details from source project ({project_id})"
    )
    name = f"projects/{project_id}/locations/{location}/processors/{processor_id}"
    request = documentai.GetProcessorRequest(
        name=name,
    )
    response = client.get_processor(request=request)
    print(f"Processor Name: {response.name}")
    print(f"Processor Display Name: {response.display_name}")
    print(f"Processor Type: {response.type_}")
    return response.display_name, response.type_

def sample_create_processor(project_id: str, location