Use auto data quality

This document describes how to use Knowledge Catalog (formerly Dataplex Universal Catalog) data quality scans to measure, monitor, and manage the quality of your data. Data quality scans help you automate the process of validating your data for completeness, validity, and consistency.

With data quality scans, you can define rules to check for missing values, ensure values match a regular expression or belong to a set, verify uniqueness, or use custom SQL for more complex validations such as anomaly detection. This document explains how to create and manage data quality scans.

Learn more about data quality scans. Learn more about reusing data quality rules across multiple scans.

Before you begin

  1. Enable the Dataplex API.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the API

  2. Optional: To generate recommendations for data quality rules based on the results of a data profile scan, create and run the data profile scan.

Required roles and permissions

This section describes the IAM roles and permissions needed to use Knowledge Catalog data quality scans.

User roles and permissions

To get the permissions that you need to run and manage data quality scans, ask your administrator to grant you the following IAM roles:

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to run and manage data quality scans. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to run and manage data quality scans:

  • Run a data quality scan on a BigQuery table:
    • bigquery.jobs.create on the project to run scan jobs
    • bigquery.tables.get on the BigQuery table to be scanned
    • bigquery.tables.getData on the BigQuery table to be scanned
  • Publish data quality scan results to Knowledge Catalog:
    • bigquery.tables.update on the scanned table
    • dataplex.entryGroups.useDataQualityScorecardAspect on the @bigquery entry group in the same location as the table
  • Create a DataScan: dataplex.datascans.create on the project
  • Delete a DataScan: dataplex.datascans.delete on the project
  • View DataScan metadata: dataplex.datascans.get on the project
  • View DataScan details including rules and results: dataplex.datascans.getData on the project
  • List DataScans: dataplex.datascans.list on the project
  • Run a DataScan: dataplex.datascans.run on the project
  • Update a DataScan: dataplex.datascans.update on the project
  • Get or set IAM policy on a DataScan:
    • dataplex.datascans.getIamPolicy on the project
    • dataplex.datascans.setIamPolicy on the project

You might also be able to get these permissions with custom roles or other predefined roles.

If you need to access columns protected by BigQuery column-level access policies, then you also need permissions for those columns.

Knowledge Catalog service account roles and permissions

If you haven't created any data quality or data profile scans or you don't have a Knowledge Catalog lake in this project, create a service identifier by running: gcloud beta services identity create --service=dataplex.googleapis.com. This command returns a Knowledge Catalog service identifier if it exists.

To ensure that the Knowledge Catalog service account of the project containing the data quality scan has the necessary permissions to read data from various sources and export results, ask your administrator to grant the following IAM roles to the Knowledge Catalog service account of the project containing the data quality scan:

  • Read BigQuery table data: BigQuery Data Viewer (roles/bigquery.dataViewer) on BigQuery tables to be scanned and any other tables referenced in rules
  • Read Iceberg REST Catalog table data: BigLake Viewer (roles/biglake.viewer) on Iceberg REST Catalog tables to be scanned and any other tables referenced in rules
  • Export scan results to a BigQuery table: BigQuery Data Editor (roles/bigquery.dataEditor) on the results dataset and table
  • Scan BigQuery data organized in a Knowledge Catalog lake:
  • Scan a BigQuery external table from Cloud Storage: Storage Object Viewer (roles/storage.objectViewer) on the Cloud Storage bucket

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to read data from various sources and export results. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to read data from various sources and export results:

  • Read BigQuery table data:
    • bigquery.tables.get on BigQuery tables
    • bigquery.tables.getData on BigQuery tables
  • Export scan results to a BigQuery table:
    • bigquery.datasets.get on results dataset and table
    • bigquery.tables.create on results dataset and table
    • bigquery.tables.get on results dataset and table
    • bigquery.tables.getData on results dataset and table
    • bigquery.tables.update on results dataset and table
    • bigquery.tables.updateData on results dataset and table
  • Scan BigQuery data organized in a Knowledge Catalog lake:
    • dataplex.lakes.list on Dataplex resources
    • dataplex.lakes.get on Dataplex resources
    • dataplex.zones.list on Dataplex resources
    • dataplex.zones.get on Dataplex resources
    • dataplex.entities.list on Dataplex resources
    • dataplex.entities.get on Dataplex resources
    • dataplex.operations.get on Dataplex resources
  • Scan a BigQuery external table from Cloud Storage:
    • storage.buckets.get on the Cloud Storage bucket
    • storage.objects.get on the Cloud Storage bucket

Your administrator might also be able to give the Knowledge Catalog service account of the project containing the data quality scan these permissions with custom roles or other predefined roles.

If you need to access columns protected by BigQuery column-level access policies, then assign the Knowledge Catalog service account permissions for those columns.

If a table has BigQuery row-level access policies enabled, then you can only scan rows visible to the Knowledge Catalog service account. Note that the individual user's access privileges are not evaluated for row-level policies.

Networking requirements

To run a scan, you must enable Private Google Access on the VPC subnet that you use for the scan. If you don't specify a subnet, make sure your default subnet has Private Google Access enabled.

Define data quality rules

You can define data quality rules by using the following:

If you're using the Google Cloud CLI, you can define these rules in a JSON or YAML file.

The examples in the following sections show how to define a variety of data quality rules. The rules validate a sample table that contains data about customer transactions. Assume the table has the following schema:

Column name Column type Column description
transaction_timestamp Timestamp Timestamp of the transaction. The table is partitioned on this field.
customer_id String A customer ID in the format of 8 letters followed by 16 digits.
transaction_id String The transaction ID needs to be unique across the table.
currency_id String One of the supported currencies. The currency type must match one of the available currencies in the dimension table dim_currency.
amount float Transaction amount.
discount_pct float Discount percentage. This value must be between 0 and 100.

Define data quality rules using built-in rule types

The following examples are based on built-in rule types. You can create rules based on built-in rule types using the Google Cloud console or the API. Knowledge Catalog might recommend some of these rules.

Column name Rule Type Suggested dimension Rule parameters
transaction_id Uniqueness check Uniqueness Threshold: Not Applicable
amount Null check Completeness Threshold: 100%