As of April 10, 2026, Dataplex Universal Catalog is now called Knowledge Catalog. The API, client library, CLI, and IAM names remain unchanged. For more information, see Introducing the Google Cloud Knowledge Catalog.

About data lineage

Data lineage is a visual map that tracks the entire lifecycle of your data. It shows you where your data comes from (the origin), where it travels (the destinations), and all the changes or transformations that happen along the way.

You can view this complete map of your data's journey directly in the Google Cloud console for assets created in products such as Knowledge Catalog (formerly Dataplex Universal Catalog), BigQuery (including external tables created for Iceberg REST Catalog), and Vertex AI. Because workflows often span multiple regions, Knowledge Catalog supports multi-region lineage, which provides a unified view of your data's journey across the global Google Cloud ecosystem. Advanced users can also retrieve this information by using the Data Lineage API.

Why you need data lineage

Modern companies move and change large amounts of data constantly. For example, transforming raw customer purchases into reports, dashboards, and machine learning models. This complexity creates critical challenges for your team:

Trust and verification. Data users often struggle to confirm that the reports and numbers they see are accurate and come from a trusted source.
Troubleshooting. When an error appears in a final report, data teams might find it difficult and time-consuming to trace the issue back through every step to its root cause.
Change management. Before changing or deleting a piece of data (such as a column in a table), teams need to know every single downstream report or model that relies on it to avoid breaking critical systems.
Compliance. Leaders need visibility into how sensitive data (such as customer or financial information) is used across the organization to meet regulatory requirements.

Data lineage solves these problems by providing a clear, visual, and documented journey of your data. This lets you understand data sources, trace errors, assess the impact of changes, and maintain compliance.

How data lineage works

The data lineage workflow includes the following steps:

Data sources and ingestion: lineage information from your data sources initiates the entire process.
- Google Cloud services: when the Data Lineage API is enabled, supported services such as BigQuery and Dataflow automatically report lineage events whenever data is moved or transformed.
- Custom sources: for any systems not automatically supported by Google Cloud integrations, you can use the Data Lineage API to manually record lineage information. We recommend importing events formatted according to the OpenLineage standard.
Lineage platform: this central platform ingests, models, and stores all lineage data.
- Data Lineage API: this API acts as the single entry point for all incoming lineage information. It uses a hierarchical data model consisting of three core concepts: process, run, and event.
- Processing and storage: the platform processes incoming data and stores it in reliable, query-optimized databases.
User experience: you can interact with the stored lineage information in two primary ways:
- Visual exploration: in the Google Cloud console, a frontend service fetches and renders the lineage data as an interactive graph or list. This is supported for Knowledge Catalog, BigQuery, Lakehouse for Apache Iceberg (for Iceberg REST Catalog tables), physical layer (Cloud Storage), and Vertex AI (for models, datasets, through pipelines; and feature store views, and feature groups). This is ideal for visually exploring your data's journey.
- Programmatic access: using an API client, you can directly communicate with the Data Lineage API to automate lineage management. This lets you write lineage information from custom sources. It also lets you read and query the stored lineage data for use in other applications or for building custom reports.
Note: For historical reasons, the data lineage client libraries and API packages (for example, google-cloud-datacatalog-lineage in Python) use the datacatalog name in their identifiers. However, data lineage is independent of the Data Catalog service and remains supported in Knowledge Catalog.

Which method should I use for data lineage?

To perform immediate, single-level lookups, use the SearchLinks method. To build a complete lineage graph or perform deep impact analysis (up to 100 levels), use the SearchLineageStreaming method.

Depending on your use case, select the most appropriate method:

Feature	`SearchLinks`	`SearchLineageStreaming`
Depth	1 level (immediate neighbors)	Up to 100 levels
Execution	Synchronous	Real-time streaming
Use case	Simple lookups of direct sources or targets	Building a complete lineage graph or performing impact analysis

Identify direction

Upstream (Origins):
- In SearchLinks, set the target field to your asset's FQN.
- In SearchLineageStreaming, set direction to UPSTREAM.
Downstream (Destinations):
- In SearchLinks, set the source field to your asset's FQN.
- In SearchLineageStreaming, set direction to DOWNSTREAM.

Data lineage information model

Lineage is a record of data being transformed from sources to targets. The Data Lineage API collects this information and organizes it into a hierarchical data model that uses the concepts of processes, runs, and events.

Concept	Description
Process	A data transformation definition.
Run	An execution of a process.
Event	A record of data movement during a run.

What is a lineage process?

A process is the definition of a data transformation operation for a specific system. For BigQuery lineage, a process is a job of a supported job type. All executions of the same SQL query are linked to a single process, which lets you track every instance where a specific transformation logic is used.

For example, the following SQL query is a process. This query creates a table by counting the total number of trips for each vendor from two source tables.

  CREATE TABLE `dataplex-docs.data_lineage_demo.total_green_trips_22_21`
  AS
  SELECT
      vendor_id,
      COUNT(*) AS number_of_trips
  FROM
      (
          SELECT vendor_id
          FROM `dataplex-docs.data_lineage_demo.nyc_green_trips_2022`
          UNION ALL
          SELECT vendor_id
          FROM `dataplex-docs.data_lineage_demo.nyc_green_trips_2021`
      )
  GROUP BY
      vendor_id;

The REST resource name format for a process is projects/PROJECT_NUMBER/locations/LOCATION/processes/PROCESS_ID.

For example: projects/123456789123/locations/us/processes/sh-0548bbf4ff3c8072a6c7372ba1acafb6

For more information about the process resource, see the Process resource reference.

What is a lineage run?

A run is a single execution of a process. Processes can have multiple runs.

Each run is a unique operation characterized by a startTime, an endTime, and a final state, such as COMPLETED, FAILED, or ABORTED.

For example, executing the SQL query from the Process section at 9:00 AM creates a specific run. Executing the same query again at 10:00 AM creates a new, distinct run. Both runs are linked to the same parent process.

The REST resource name format for a run shows that it's a child of a process: projects/PROJECT_NUMBER/locations/LOCATION/processes/PROCESS_ID/runs/RUN_ID.

For example: projects/123456789123/locations/us/processes/sh-0548bbf4ff3c8072a6c7372ba1acafb6/runs/83dd03a51cd2ac80f465c9e267a950b1

For more information about the run resource, see the Run resource reference.

What is a lineage event?

An event represents a point in time when a data transformation moves data between a source and a target entity. An event is a granular record of a specific data movement that connects source and target tables for a specific run. An event can also have multiple sources and targets.

For example, if your run executes the SQL query discussed in the Process section, a lineage event records that the nyc_green_trips_2021 and nyc_green_trips_2022 source tables are used to create the total_green_trips_22_21 target table.

A lineage event contains a list of links that define the source and target. Events are used to create lineage graphs. Although the Google Cloud console presents these lineage graphs, it doesn't directly display individual events. You can create, read, and delete, but not update events by using the Data Lineage API.

Each link within an event defines a single path of data flow from a source entity to a target entity. An entity is a reference to a data asset, such as a BigQuery table, and is identified by its Fully Qualified Name (FQN). A single event can contain multiple links, which is common in operations such as table joins where multiple sources contribute to one target.

For details on how events support column-level lineage, see Column-level lineage.

What data sources are supported for data lineage?

You can populate lineage information in Knowledge Catalog in the following ways:

Automatically from integrated Google Cloud services
Manually, by using the Data Lineage API for custom sources
By importing events from OpenLineage

BigQuery

When you enable data lineage in your BigQuery project, Knowledge Catalog automatically records lineage information for the following:

New tables created as a result of the following BigQuery jobs:
- Copy jobs
- Load jobs that use a Cloud Storage URI
- Query jobs that use the following data definition language (DDL) in GoogleSQL:
  - CREATE TABLE
  - CREATE TEMP TABLE
  - CREATE TABLE AS SELECT
  - CREATE TABLE COPY
  - CREATE TABLE CLONE
  - CREATE TABLE FUNCTION
  - CREATE TABLE LIKE
  - CREATE VIEW
  - CREATE MATERIALIZED VIEW
Existing tables when you use the following data manipulation language (DML) statements in GoogleSQL:
- SELECT in relation to any of the listed table types:
- INSERT SELECT
- MERGE
- UPDATE
- DELETE

BigQuery copy, query, and load jobs are represented as processes.

To view the process details, on the lineage graph, click the Process details icon .

Each process contains the BigQuery