Export metadata

This document explains how to export metadata from Knowledge Catalog (formerly Dataplex Universal Catalog) using a metadata export job for use in external systems. You can export metadata to analyze it with tools such as BigQuery, programmatically process large volumes, or integrate it with custom applications.

This guide assumes you are familiar with Knowledge Catalog metadata concepts, including entry groups, entry types, and aspect types.

Job scope

The job scope defines the metadata to export. You must provide one of the following job scopes for each metadata export job:

• Organization: exports the metadata that belongs to your organization.
• Projects: exports the metadata that belongs to the specified projects.
• Entry groups: exports the metadata that belongs to the specified entry groups.

You can further restrict the scope by specifying the entry types or aspect types to include in the job. The job exports only the entries and aspects that belong to these entry types and aspect types.

VPC Service Controls

Knowledge Catalog uses VPC Service Controls to provide additional security for metadata export jobs. The project that the job belongs to determines the VPC Service Controls perimeter, as follows:

• If you set the job scope to the organization level, the following things happen:
  • The export scope is the organization that the job belongs to.
  • Only the entries that are within the VPC Service Controls perimeter are exported.
  • Any projects that are within the job's organization but outside the VPC Service Controls perimeter are excluded.
• If you set the job scope to projects or entry groups, the projects or entry groups must be in the same VPC Service Controls perimeter as the job. If any of the projects or entry groups violate VPC Service Controls rules, the job fails.

Before you begin

Before you export metadata, complete the tasks in this section.

Required roles for end users

To get the permissions that you need to manage metadata export jobs, ask your administrator to grant you the following IAM roles:

• Create metadata export jobs:
  • Dataplex Entry Group Exporter (roles/dataplex.entryGroupExporter) on the organization, the projects, or the entry groups to export
  • Dataplex Metadata Job Owner (roles/dataplex.metadataJobOwner) on the project that you run the metadata job in
• Access the exported results: Storage Object Viewer (roles/storage.objectViewer) on the project or the bucket
• View metadata jobs: Dataplex Metadata Job Viewer (roles/dataplex.metadataJobViewer) on the project

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to manage metadata export jobs. To see the exact permissions that are required, expand the Required permissions section:

You might also be able to get these permissions with custom roles or other predefined roles.

Required roles for the Knowledge Catalog service account

To ensure that the Knowledge Catalog service account has the necessary permissions to access the Cloud Storage bucket, ask your administrator to grant the Knowledge Catalog service account the following permissions on the bucket: storage.buckets.get, storage.objects.get, and storage.objects.create.

Configure Google Cloud resources

1. Install the Google Cloud CLI. After installation, initialize the Google Cloud CLI by running the following command:

   gcloud init

   If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

2. Create a Cloud Storage bucket to store the exported results.

   The bucket must be in the same location and the same VPC Service Controls perimeter as the metadata job.

Run a metadata export job

The following sections show how to export metadata with different job scopes.

Export metadata from your organization

To export the metadata from your organization, use the metadataJobs.create method and set the organizationLevel boolean to true.

Before using any of the request data, make the following replacements:

• JOB_PROJECT: the Google Cloud project that you run the metadata job in. Provide a project number or project ID.
• LOCATION_ID: the Google Cloud location, such as us-central1.
• METADATA_JOB_ID: optional. The metadata job ID.

• BUCKET: the Cloud Storage bucket to export the metadata to.

  Optionally, you can include a custom prefix after the bucket name, in the format gs://BUCKET/PREFIX/. The maximum length of the custom prefix is 128 characters.

HTTP method and URL:

POST https://dataplex.googleapis.com/v1/projects/JOB_PROJECT/locations/LOCATION_ID/metadataJobs?metadataJobId=METADATA_JOB_ID

Request JSON body:

{
  "type": "EXPORT",
  "export_spec": {
    "output_path": "gs://BUCKET/",
    "scope": {
      "organizationLevel": true,
    },
  }
}

To send your request, expand one of these options:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://dataplex.googleapis.com/v1/projects/JOB_PROJECT/locations/LOCATION_ID/metadataJobs?metadataJobId=METADATA_JOB_ID"

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://dataplex.googleapis.com/v1/projects/JOB_PROJECT/locations/LOCATION_ID/metadataJobs?metadataJobId=METADATA_JOB_ID" | Select-Object -Expand Content

The response identifies a long-running operation. The exported metadata is saved to a Cloud Storage bucket.

Export metadata from specific projects

To export metadata from one or more projects, use the metadataJobs.create method and provide a list of projects.

Before using any of the request data, make the following replacements:

• JOB_PROJECT: the Google Cloud project that you run the metadata job in. Provide a project number or project ID.
• LOCATION_ID: the Google Cloud location, such as us-central1.
• METADATA_JOB_ID: optional. The metadata job ID.

• BUCKET: the Cloud Storage bucket to export the metadata to.

  Optionally, you can include a custom prefix after the bucket name, in the format gs://BUCKET/PREFIX/. The maximum length of the custom prefix is 128 characters.

• METADATA_SOURCE_PROJECT: a project whose metadata you want to export. Provide a project number or project ID. The project must be in the same organization and VPC Service Controls perimeter as the metadata job.

HTTP method and URL:

POST https://dataplex.googleapis.com/v1/projects/JOB_PROJECT/locations/LOCATION_ID/metadataJobs?metadataJobId=METADATA_JOB_ID

Request JSON body:

{
  "type": "EXPORT",
  "export_spec": {
    "output_path": "gs://BUCKET/",
    "scope": {
      "projects": [
        "projects/METADATA_SOURCE_PROJECT",
        # Additional projects
      ],
    },
  }
}

To send your request, expand one of these options:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \