Cloud Storage Ingest pipeline

This document describes what the Cloud Storage ingest pipeline does and how to run the pipeline.

What does the Cloud Storage ingest pipeline do?

Your users can transfer documents from Cloud Storage to the Document AI Warehouse. There, users can opt to have them processed for tasks such as searching, document management workflows, or simply testing out Document AI outputs.

Why use this pipeline?

If many documents must be ingested (with or without processing), this pipeline provides a reliable workflow. It also helps users accelerate time to onboard Document AI Warehouse customers for proofs of concept or production workloads.

Pipeline features

In general, the Cloud Storage ingest pipeline supports the following actions:

  • Ingest raw documents into Document AI Warehouse.

    {
  name: "projects/PROJECT_NUMBER/locations/LOCATION_ID",
  gcs_ingest_pipeline: {
    input_path: "gs://BUCKET_NAME/FOLDER_NAME",
    schema_name: "projects/PROJECT_NUMBER/locations/LOCATION_ID/documentSchemas/DOCUMENT_SCHEMA_ID"
  }
}

  • Ingest Document AI processed documents into Document AI Warehouse.

    {
  name: "projects/PROJECT_NUMBER/locations/LOCATION_ID",
  gcs_ingest_pipeline: {
    input_path: "gs://BUCKET_NAME/FOLDER_NAME",
    schema_name: "projects/PROJECT_NUMBER/locations/LOCATION_ID/documentSchemas/DOCUMENT_SCHEMA_ID",
    processor_type: "PROCESS_TYPE"
  }
}

    The processor_type is required to indicate that the input files have been processed by Document AI. processor_type (OCR_PROCESSOR, INVOICE_PROCESSOR, FORM_PARSER_PROCESSOR) can be found under the Type in API field here.

  • Ingest Document AI processed documents and corresponding raw documents in the same request.