Migrate metadata from Dataproc Metastore to Lakehouse

This document explains how to migrate metadata from a Dataproc Metastore service to an Apache Iceberg REST catalog endpoint or a Hive catalog endpoint, built on borderless Lakehouse.

Use cases

• Serverless Modernization: Transition from a conventional Hive Metastore (HMS) to an automatically scaling, fully managed catalog, which eliminates the operational overhead of metastore management.
• Multi-Engine Collaboration: Enable data sharing across engines—including Apache Spark, Apache Flink, Apache Hive, and BigQuery—so data scientists and analysts can work on the same tables simultaneously without file duplication.
• Direct BigQuery Integration: Query open-source tables directly from BigQuery with high-performance execution.
• Unified Governance: Consolidate metadata into a single source of truth for simplified data discovery and consistent policy enforcement.
• Modern Table Formats: Seamlessly adopt advanced open formats like Apache Iceberg while maintaining full compatibility with your existing Hive workloads.

Before you begin

1. Make sure that an active Dataproc Metastore service exists as the migration source.
2. Make sure that the target Hive catalog or Iceberg catalog exists and includes the Cloud Storage buckets or paths where your source table data and metadata reside (for example, the Dataproc Metastore warehouse bucket, such as gs://gcs-your-project-name-0825d7b3-0627-4637-8fd0-cc6271d00eb4/hive-warehouse).

   If the destination catalog doesn't include the data location, table migration fails because the target catalog cannot register the tables. For Iceberg catalog creation, see Set up the Iceberg REST catalog endpoint.

   To create a Hive catalog, see Create a Lakehouse Hive catalog.
Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Verify that billing is enabled for your Google Cloud project.

Enable the Lakehouse for Apache Iceberg, Dataproc Metastore APIs.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

Verify that billing is enabled for your Google Cloud project.

Enable the Lakehouse for Apache Iceberg, Dataproc Metastore APIs.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

Required roles

To get the permissions that you need to trigger the migration, ask your administrator to grant you the following IAM roles on the Dataproc Metastore service:

• Start the migration: Dataproc Metastore Editor (roles/metastore.editor)
• Create Hive or Iceberg catalogs: BigLake Admin (roles/biglake.admin)
• Migrate metadata to destination catalogs using a target project: BigLake Admin (roles/biglake.admin) on the Dataproc Metastore service agent (service-PROJECT_NUMBER@gcp-sa-metastore.iam.gserviceaccount.com).
• Write migration reports for the report bucket (if not using the service artifacts bucket): Storage Object Admin (roles/storage.objectAdmin) on the Dataproc Metastore service agent (service-PROJECT_NUMBER@gcp-sa-metastore.iam.gserviceaccount.com)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

How a migration works

The migration process works as follows:

1. Choose your target catalog: Select the destination Hive catalog endpoint or Apache Iceberg REST catalog endpoint for your migration.
2. Trigger migration: Run the gcloud beta metastore services migrations start command or call the startMigration method on your Dataproc Metastore service to initiate the migration.
3. Poll for status: Monitor the progress of the migration using the gcloud beta metastore services migrations describe command or by polling the target execution.
4. Review reports: Review the detailed JSON reports written to your specified Cloud Storage path to verify the results.

Run a migration

To run a migration, you trigger the migration process and then monitor its progress.

Start the migration

To trigger the metadata migration on a Dataproc Metastore service, use the gcloud CLI or the REST API.

gcloud beta metastore services migrations start SERVICE_ID \
    --location=REGION \
    --hive-catalog="projects/PROJECT_ID/catalogs/HIVE_CATALOG_ID" \
    --hive-databases="HIVE_DB_1,HIVE_DB_2" \
    --iceberg-catalog="projects/PROJECT_ID/catalogs/ICEBERG_CATALOG_ID" \
    --iceberg-namespaces="ICEBERG_NAMESPACE_1,ICEBERG_NAMESPACE_2" \
    --async

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    -d '{
      "migrationExecution": {
        "biglakeMetastoreMigrationConfig": {
          "mode": "BACKFILL",
          "dryRun": false,
          "reportPath": "gs://BUCKET_NAME/PATH/",
          "conflictPolicy": "SKIP",
          "hiveConfig": {
            "catalog": "projects/PROJECT_ID/catalogs/HIVE_CATALOG_ID",
            "databases": ["HIVE_DB_1", "HIVE_DB_2"]
          },
          "icebergConfig": {
            "catalog": "projects/PROJECT_ID/catalogs/ICEBERG_CATALOG_ID",
            "namespaces": ["ICEBERG_NAMESPACE_1", "ICEBERG_NAMESPACE_2"]
          }
        }
      }
    }' \
    "https://metastore.googleapis.com/v1beta/projects/PROJECT_ID/locations/REGION/services/SERVICE_ID:startMigration"

Poll the migration execution

The request starts a long-running operation (LRO) and returns a unique migration execution ID. You can monitor the progress of your run using the gcloud CLI or the REST API:

gcloud beta metastore services migrations describe MIGRATION_EXECUTION_ID \
    --service=SERVICE_ID \
    --location=REGION

curl -X GET \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    "https://metastore.googleapis.com/v1beta/projects/PROJECT_ID/locations/REGION/services/SERVICE_ID/migrationExecutions/MIGRATION_EXECUTION_ID"