Metadata is crucial for organizing and understanding your data assets and relationships between them. For example, regulated industries need to protect personally identifiable information (PII) such as names, addresses, and government issued ID numbers. This data can occur in numerous instances across distributed data systems. Knowledge Catalog (formerly Dataplex Universal Catalog) helps you quickly discover and catalog your distributed data assets. You can then classify data as being PII.
By enriching data entries and entry links with meaningful context, you can make your data more discoverable and useful. Knowledge Catalog enables context through aspect types and aspects:
- Aspect type: A JSON template defining related data. For example, for compliance information, you might have PII classification and GDPR compliance.
- Aspect: An instance of an aspect type. For example, {"pii_classification": "confidential", "gdpr_compliant": true}. You can apply an aspect to an entire table, an individual table column, or an entry link.
Once you've classified data, you can apply data quality or access policies to the data.
For more information, see About metadata management in Knowledge Catalog.
What are aspect types?
Aspect types are reusable resources that provide templates for aspects. When you create a custom aspect type, you must specify an ID that complies with Knowledge Catalog resource naming guidelines.
Categories of aspect types
Aspect types are categorized into custom and system aspect types:
Custom aspect types
Aspect types that you create in Knowledge Catalog are called custom aspect types.
Custom aspect types can be global or regional:
- Global aspect types: Can be used to create aspects for entries in any
region. Choose a
globalaspect type if you need to apply the same structure to entries across multiple regions, as you only need to define it once. Because the definition of a global aspect type is replicated across all regions, this option may not be suitable if the aspect type schema itself contains sensitive information or if strict data residency for all metadata components is required. - Regional aspect types: Can only be used to create aspects for entries
that reside in the same region as the aspect type. For example, an aspect
type created in
us-central1can only be used with entries inus-central1. Choose a regional location if an aspect type is only relevant for entries in a specific region, or to ensure that the aspect type definition resides within the selected region, which can help meet data sovereignty and compliance requirements. If you use regional aspect types but need the same structure in multiple regions, you must create and manage separate aspect type definitions in each region.
You can create custom aspect types either in a specific regional location
(for example, us-central1) or as a global resource. The location of an
aspect type impacts the scope of its applicability and determines which entries
it can be used with.
For more information, see Project and location constraints.
System aspect types
Aspect types that Knowledge Catalog provides, uses, and manages are called
system aspect types. System aspect types are always global, therefore they can
be used in any region. System aspect types are stored in a Google-managed
project with project number 655216118709. For example,
projects/655216118709/locations/global/aspectTypes/schema.
Each system-defined aspect type, entry type, and entry link type has its own IAM permissions. For more information, see IAM permissions for system aspect types, entry types, and entry link types.
System aspect types are further categorized into reusable and restricted. The following table describes the categories of system aspect types, and the list of aspect types that Knowledge Catalog provides for each of the categories:
| Category of system aspect type | Description | Aspect types that Knowledge Catalog provides |
|---|---|---|
| Reusable system aspect type | You can use these aspect types to create or modify aspects. |
|
| Restricted system aspect type | Knowledge Catalog manages these aspect types. You can read aspects under these aspect types, but can't create or edit aspects under these aspect types. |
Including but not limited to:
|
What are aspects?
Aspects let you capture metadata within entries or entry links to provide meaningful context. You can use aspects to store the following types of metadata:
Business metadata: Information that provides business context, such as data classification.
Technical metadata: Technical details of entries or entry links, including the following:
- For entries: details about the data asset itself, for example, its schema.
- For entry links: details about the relationship between data assets, for example, join details.
Data-derived metadata: Information generated from the data itself, such as statistics from a BigQuery table.
Aspects are component parts of the parent entry resource rather than separate standalone resources. Modifying an aspect updates the containing entry. To track these modifications, you can subscribe to Metadata change feeds.
Aspect placement and target levels
Every aspect is an instance of an aspect type template, which defines the fields you must populate. Where you can apply an aspect and how many times it can appear depends on the target resource type:
| Target resource | Supported levels | Cardinality limits |
|---|---|---|
| Entries | Entry-level or column-level | Maximum of one aspect per aspect type at the entry level. Multiple aspects per aspect type are allowed across different columns. |
| Entry links | Link-level only | Maximum of one aspect per aspect type per link. |
Categories of aspects
Aspects are categorized into required aspects and optional aspects:
Required aspects
Required aspects are the aspects that are mandatory upon creation of an entry or entry link. Such aspects are defined by the entry type or entry link type. All resources belonging to a specific type must always have all of the required aspects that are defined by that type.
Knowledge Catalog manages the required aspects (for example, schema) for system entries.
Note the following:
You can associate required aspects only with entries and not with its columns.
You can't delete the required aspects from an entry or entry link.