Data lineage helps you understand how data moves through your systems by tracking the relationships between data assets and the processes that transform them. You can view this lineage information as graphs and lists in the Google Cloud console.
This document describes table-level and column-level data lineage granularity, and provides instructions for using graph and list views to explore data lineage in the Google Cloud console.
For details on the underlying data model, see the data lineage information model.
Differences between table-level and column-level lineage
Data lineage lets you trace the origin and transformation path of your data at both the table and column levels.
When to use table-level lineage
Table-level lineage provides a high-level overview of your data pipelines by showing the relationships between entire tables. Use table-level lineage for macro-level tasks such as the following:
Data discovery. An analyst building a new dashboard can use table-level lineage to trace a summary table back to its sources and confirm that the data originates from an authoritative database.
Migration planning. A database administrator planning to migrate a core database can use table-level lineage to identify every downstream report and dashboard that depends on it.
Auditing and governance. A data governor can use table-level and column-level lineage to check how data from a table that contains personally identifiable information (PII) flows through a pipeline.
When to use column-level lineage
Column-level lineage provides a more granular view by tracking the flow of data between individual columns. In this view, the links within a lineage event represent the relationship between a source column and a target column. Each of these column-level links has a dependency type that describes the transformation:
Exact copy: values are copied between columns.Other: other types of dependencies between columns.
Use column-level lineage for tasks such as the following:
Root cause analysis. If a data analyst finds an incorrect value in a column, they can use column-level lineage to trace it back to the source columns to find the root cause.
Impact analysis. Before a data engineer deprecates a column, they can use column-level lineage to find every downstream column that depends on it.
Data source verification for metrics. A data analyst can use column-level lineage to identify which source columns are used to calculate a metric without deciphering a complex SQL query.
Column-level lineage is automatically collected for the following types of BigQuery jobs:
For Managed Service for Apache Spark jobs, support depends on the type and version of the Open Lineage dependency used by Managed Service for Apache Spark. The minimum supported version is 1.34. The following are the minimum supported Managed Service for Apache Spark cluster image versions:
- 3.0.3
- 2.3.22
- 2.2.75
- 2.1.107
The following are the minimum supported Managed Service for Apache Spark runtime versions:
- 3.0.3
- 2.3.20
Lineage views in the Google Cloud console
Data lineage in the Google Cloud console lets you interact with lineage information in two ways: you can explore the lineage graph across multiple available regions, or you can use the Lineage explorer panel to get a more focused view within a specific region. You can also switch between the Graph view and the List view to analyze data flow at different levels of detail.
Lineage views are available only for Knowledge Catalog (formerly Dataplex Universal Catalog) entries, BigQuery assets, and Vertex AI resources (models, datasets, feature store views, and feature groups).
To see the different views discussed in this document, see Use data lineage with Google Cloud systems.
Lineage graph view
The Graph view visualizes data asset flow and relationships across systems and regions, helping you understand data architecture, trace origins and destinations, and identify patterns. These lineage graphs, generated by the Data Lineage API service for a specific Knowledge Catalog entry, show how data is transformed over time, displaying upstream, downstream, or both flows from a selected root entry.
The Data Lineage API automatically receives asset information from supported systems and through API calls for custom sources.
The key elements in the graph are as follows:
Nodes. Nodes represent the data entities. In a table-level view, a node shows the table name and its columns. In a column-level view, each node represents a specific table and column.
Edges. Edges are the lines that connect nodes and represent the processes that occur between them. The appearance of an edge depends on the lineage view:
- In the table-level view, edges have icons to indicate data transformations.
- In the column-level view, edges have labels to indicate data
transformations. For example, an edge label might say
Exact copyto describe how a source column was copied to a target column.
Process icons and labels. Process icons and labels appear on edges to provide more information about the transformation.
- Icons. Icons represent the transformation process. When you manually explore the graph, icons on edges represent the source system of the process (for example, BigQuery or Vertex AI). If multiple processes are involved, a 'multiple processes' icon is displayed. If the process source system is unknown, a gear icon is used. When you apply filters, a gear icon is used for all processes.
- Labels. In the column-level lineage view, a label describes the type
of dependency between columns:
Exact copyorOther.
Explore the lineage graph
When you open the Lineage tab, you see the default Graph view. The default view provides a high-level overview across systems and regions, with manual and incremental graph expansion that can load five nodes at a time. Process icons on edges represent the source system or indicate multiple processes.
