Integrate with OpenLineage

This document explains how to integrate OpenLineage with Knowledge Catalog (formerly Dataplex Universal Catalog) to import and visualize data lineage from external systems. By acting as an OpenLineage consumer using the ProcessOpenLineageRunEvent REST API, Knowledge Catalog lets you unify custom pipeline lineage alongside built-in lineage from Google Cloud services.

Overview

OpenLineage is an open platform for collecting and analyzing data lineage information. Using an open standard for lineage data, OpenLineage captures lineage events from data pipeline components which use an OpenLineage API to report on runs, jobs, and datasets.

Through the Data Lineage API, you can import OpenLineage events to display in the Knowledge Catalog web interface alongside lineage information from Google Cloud services, such as BigQuery, Managed Service for Apache Airflow, Cloud Data Fusion, and Managed Service for Apache Spark.

To import OpenLineage events that use the OpenLineage specification, use the ProcessOpenLineageRunEvent REST API method, and map OpenLineage facets to Data Lineage API attributes.

OpenLineage integration limitations

  • Supported versions: The Data Lineage API supports OpenLineage major version 1.

  • API actions: The Data Lineage API endpoint ProcessOpenLineageRunEvent only acts as a consumer of OpenLineage messages, not a producer. The API lets you send lineage information generated by any OpenLineage-compliant tool or system into Knowledge Catalog. Some Google Cloud services, such as Managed Service for Apache Spark and Managed Airflow, include built-in OpenLineage producers that can send events to this endpoint, automating lineage capture from those services.

  • Unsupported features: The Data Lineage API doesn't support the following:

    • Any subsequent OpenLineage release with message format changes
    • DatasetEvent
    • JobEvent

  • Message size: Maximum size of a single message is 5 MB.

  • Name length: Length of each Fully Qualified Name in inputs and outputs is limited to 4000 characters.

  • Link limits: Links are grouped by events, with a maximum of 100 links per event. The maximum aggregate number of table-level links is 1000. If a message contains more than 1500 column-level links, the column-level information is skipped.

  • Graph scope: Knowledge Catalog displays a lineage graph for each job run, showing the inputs and outputs of lineage events. It doesn't support lower-level processes such as Spark stages.

OpenLineage facet attribute mapping

For information about OpenLineage mapping, see OpenLineage mapping.

Import an OpenLineage event

If you haven't yet set up OpenLineage, see Getting started.

To import an OpenLineage event into Knowledge Catalog, call the API method ProcessOpenLineageRunEvent.

C#

C#

Before trying this sample, follow the C# setup instructions in the Data Lineage quickstart using client libraries. For more information, see the Data Lineage C# API reference documentation.

To authenticate to Data Lineage, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 

using Google.Cloud.DataCatalog.Lineage.V1;
using Google.Protobuf.WellKnownTypes;

public sealed partial class GeneratedLineageClientSnippets
{
    /// <summary>Snippet for ProcessOpenLineageRunEvent</summary>
    /// <remarks>
    /// This snippet has been automatically generated and should be regarded as a code template only.
    /// It will require modifications to work:
    /// - It may require correct/in-range values for request initialization.
    /// - It may require specifying regional endpoints when creating the service client as shown in
    ///   https://cloud.google.com/dotnet/docs/reference/help/client-configuration#endpoint.
    /// </remarks>
    public void ProcessOpenLineageRunEventRequestObject()
    {
        // Create client
        LineageClient lineageClient = LineageClient.Create();
        // Initialize request argument(s)
        ProcessOpenLineageRunEventRequest request = new ProcessOpenLineageRunEventRequest
        {
            Parent = "",
            OpenLineage = new Struct(),
        };
        // Make the request
        ProcessOpenLineageRunEventResponse response = lineageClient.ProcessOpenLineageRunEvent(request);
    }
}

Go

Go

Before trying this sample, follow the Go setup instructions in the Data Lineage quickstart using client libraries. For more information, see the Data Lineage Go API reference documentation.

To authenticate to Data Lineage, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 


//go:build examples

package main

import (
	"context"

	lineage "cloud.google.com/go/datacatalog/lineage/apiv1"
	lineagepb "cloud.google.com/go/datacatalog/lineage/apiv1/lineagepb"
)

func main() {
	ctx := context.Background()
	// This snippet has been automatically generated and should be regarded as a code template only.
	// It will require modifications to work:
	// - It may require correct/in-range values for request initialization.
	// - It may require specifying regional endpoints when creating the service client as shown in:
	//   https://pkg.go.dev/cloud.google.com/go#hdr-Client_Options
	c, err := lineage.NewClient(ctx)
	if err != nil {
		// TODO: Handle error.
	}
	defer c.Close()

	req := &lineagepb.ProcessOpenLineageRunEventRequest{