Enable policy-compliant lakehouse access with Knowledge Catalog

When you store your data across different storage systems, managing fragmented security can become a major challenge. You want to make sure sensitive information, such as financial records, stays protected, even if you store it in open formats like Apache Iceberg on Google Cloud storage.

Crucially, your security measures must work seamlessly across various query engines, like BigQuery SQL and Apache Spark. In this architecture, Knowledge Catalog solves this by functioning as a universal context engine. It holds the metadata needed to ensure that your security context is applied consistently, no matter which engine queries the data.

In this tutorial, you build a secure data lakehouse to solve these challenges. Using scripts, you define security policies and see Knowledge Catalog (formerly Dataplex Universal Catalog) and Lakehouse for Apache Iceberg work together to enforce the policies across different query engines.

Architecture overview

To set up fine-grained access control on an open table format like Apache Iceberg, you must create a strict, unified security architecture.

With this design, Knowledge Catalog acts as the central control plane. Knowledge Catalog dynamically enforces your security context across all supported engines.

This tutorial relies on two main concepts to create this unified architecture:

• Secure architectural layers: Instead of letting users or query engines directly access your Cloud Storage buckets, you build a secure, layered foundation based on the following attributes:
  • Open format with managed metadata: Your data stays in its open Apache Iceberg (Parquet) format inside Cloud Storage, while Lakehouse for Apache Iceberg manages the table metadata.
  • Logical security boundary: You decouple storage permissions from data queries using a secure Cloud resource connection. You never grant end users direct access to the files.
  • Compute delegation: To prevent query engines from bypassing your rules, you route all data requests through the BigQuery Storage API. Routing requests through the BigQuery Storage API ensures that the Knowledge Catalog can intercept and enforce policies even for external processing engines.
• Centralized policy enforcement: With a secure foundation in place, you use Knowledge Catalog to universally apply your rules:
  • Define once, enforce everywhere: You define policy tags in Knowledge Catalog just once, and the platform applies consistent masking rules across all your supported query engines.
  • Dynamic data masking: The system evaluates user identity during queries. Authorized users see raw values, while restricted users receive NULL outputs across all query engines.
  • Automated data lineage: Knowledge Catalog tracks data transformations automatically, creating an audit trail without custom logging code.

Objectives

• Create Apache Iceberg tables managed by BigQuery. Lakehouse manages the Iceberg metadata.
• Set up central security rules using policy tags to mask and protect sensitive columns.
• Separate physical storage permissions from logical data queries using a Cloud resource connection.
• Route queries securely through Managed Service for Apache Spark so that external engines can't bypass your security rules.
• Explore an interactive map of your data using data lineage.

Before you begin

Before you begin, do the following:

• Select a Google Cloud project for this tutorial.
• Confirm that billing is enabled for your project.

Prepare your environment

This tutorial uses Cloud Shell, a command-line environment that runs in the cloud.

1. From the Google Cloud Console, click the Cloud Shell icon on the top right toolbar.

2. Set your project variables:

   export PROJECT_ID=$(gcloud config get-value project)
   export REGION="us-central1"
   export ICEBERG_BUCKET="iceberg-retail-demo-${PROJECT_ID}"
   export DATASET_ID="lakehouse_retail_demo"
   export CONN_NAME="iceberg-bq-conn-demo"
   

3. Define variables for two user personas, a retail analyst and a retail manager:

   export USER_ANALYST="retail-analyst-demo"
   export EMAIL_ANALYST="${USER_ANALYST}@${PROJECT_ID}.iam.gserviceaccount.com"
   
   export USER_MANAGER="retail-manager-demo"
   export EMAIL_MANAGER="${USER_MANAGER}@${PROJECT_ID}.iam.gserviceaccount.com"
   export CURRENT_USER=$(gcloud config get-value account)
   

4. Enable the required Google Cloud APIs.

   gcloud services enable \
     bigquery.googleapis.com \
     bigqueryconnection.googleapis.com \
     datacatalog.googleapis.com \
     bigquerydatapolicy.googleapis.com \
     datalineage.googleapis.com \
     dataplex.googleapis.com \
     dataproc.googleapis.com \
     storage-component.googleapis.com
   

Download the tutorial source code

Download the Python scripts for this tutorial from the Google Cloud DevRel repository:

# Shallow clone without full history
git clone --depth 1 --filter=blob:none --sparse https://github.com/GoogleCloudPlatform/devrel-demos.git
cd devrel-demos

# Download only the specific folder
git sparse-checkout set data-analytics/governed-lakehouse
cd data-analytics/governed-lakehouse

Create a storage bucket

Create a new bucket to hold Iceberg table files:

gcloud storage buckets create gs://${ICEBERG_BUCKET} --location=${REGION}

Prepare identities and security

In this step, you set up compute delegation by creating a Cloud resource connection. This connection acts as a secure, delegated identity that BigQuery uses to manage and read your Iceberg files. This helps ensure that individual users never have direct access to your Cloud Storage bucket.

Run the following commands to create the connection, retrieve its auto-generated service account, and grant that account the permissions needed to manage your Iceberg data:

# Create the Cloud resource connection
bq mk --connection \
    --connection_type=CLOUD_RESOURCE \
    --location=${REGION} \
    ${CONN_NAME}

# Retrieve the connection's automatically generated Service Account
export BQ_CONN_SVC_ACCT=$(bq show --format=json --connection ${REGION}.${CONN_NAME} \
    | jq -r '.cloudResource.serviceAccountId')

# Grant Storage Object Admin to the connection for the Iceberg bucket
gcloud storage buckets add-iam-policy-binding gs://${ICEBERG_BUCKET} \
    --member="serviceAccount:${BQ_CONN_SVC_ACCT}" \
    --role="roles/storage.objectAdmin" \
    --quiet

Create service accounts for two personas: Analyst and Manager. The following commands set up these service accounts, permit your current user to impersonate them for testing, and grant them specific roles to run queries and view data.

echo "Creating Service Accounts..."
for USER in "${USER_ANALYST}" "${USER_MANAGER}"; do
    gcloud iam service-accounts create ${USER} --display-name="Lakehouse ${USER}"
done

echo "⏳ Waiting 15 seconds for rules to apply..."
sleep 15

echo "Granting roles to service accounts..."
for USER in "${USER_ANALYST}" "${USER_MANAGER}"; do
    EMAIL="${USER}@${PROJECT_ID}.iam.gserviceaccount.com"
    
    # Allow Cloud Shell to impersonate them for testing
    gcloud iam service-accounts add-iam-policy-binding ${EMAIL} \
        --member="user:${CURRENT_USER}" \
        --role="roles/iam.serviceAccountTokenCreator" \
        --quiet

    # Allow logical viewing of the catalog, querying, and running Dataproc jobs
    for ROLE in