Manage your Datastream resources using Knowledge Catalog

You can search for and manage your Datastream resources using Knowledge Catalog, which is a platform for storing, managing, and accessing your metadata. Use Knowledge Catalog to help you with the following:

  • Discovering, analyzing, and understanding your Datastream metadata.
  • Browsing your Datastream resources, such as streams, connection profiles and connectivity configurations.
  • Improving operational visibility for troubleshooting and managing data pipelines.
  • Establishing both near real-time and periodic mechanisms for consistent metadata ingestion with live and batch synchronization. For more information, see Synchronization modes.

Knowledge Catalog modeling

Datastream resources are modeled in Knowledge Catalog under a reserved, Knowledge Catalog-owned entry group and specific entry types.

Knowledge Catalog entity Resource identifier
Entry group @datastream
Entry type datastream-stream
Entry type datastream-connection-profile
Entry type datastream-private-connection

Types of metadata discovery

Knowledge Catalog metadata discovery is an automated process that scans connected data sources—like Datastream—to identify data assets and extract their technical metadata into the Knowledge Catalog.

This process uses both real-time and periodic synchronization to keep the catalog consistent with the source system. For Datastream, metadata for all streams, connection profiles, and private connectivity configurations is discovered by default:

Resource type Metadata retrieved
Streams
  • Name
  • Location
  • State (for example, RUNNING, PAUSED, FAILED)
  • Backfill strategy
  • Source and destination connection profile names
  • Source and destination configurations
Connection profiles
  • Name
  • Location
  • Type (for example, MYSQL, ORACLE, BIGQUERY)
  • Connectivity option
  • Hostname, port, and database name
Private connectivity configurations
  • Name
  • State
  • Connectivity method (for example, Private Service Connect or VPC peering)
  • VPC network and subnet details

Synchronization modes

Datastream uses the following synchronization modes to keep the Knowledge Catalog consistent with the source system.

  • Live synchronization: in this mode, Datastream publishes events to Pub/Sub topics when any metadata changes occur. This lets Knowledge Catalog receive near real-time updates by subscribing to these topics.
  • Batch synchronization: in this mode, Datastream periodically publishes full metadata exports (checkpoints) to designated Cloud Storage buckets, from which Knowledge Catalog ingests the data.

Knowledge Catalog supports keyword and natural language searches.

  • Keyword search lets you find resources using specific keywords, filters, and a defined syntax. For example, you might enter system=Datastream AND type=Stream to view all Datastream streams.
  • Natural language search (Preview) uses AI to understand semantic queries. It lets you find resources using everyday language, eliminating the need for complex syntax. For example, you can enter queries like List all Datastream streams with BigQuery destination.

For more information, see Search syntax for Knowledge Catalog.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector