Transformation reference

This topic covers the available de-identification techniques, or transformations, in Sensitive Data Protection.

Types of de-identification techniques

Choosing the de-identification transformation you want to use depends on the kind of data you want to de-identify and for what purpose you're de-identifying the data. The de-identification techniques that Sensitive Data Protection supports fall into the following general categories:

  • Redaction: Deletes all or part of a detected sensitive value.
  • Replacement: Replaces a detected sensitive value with a specified surrogate value.
  • Masking: Replaces a number of characters of a sensitive value with a specified surrogate character, such as a hash (#) or asterisk (*).
  • Crypto-based tokenization: Encrypts the original sensitive data value using a cryptographic key. Sensitive Data Protection supports several types of tokenization, including transformations that can be reversed, or "re-identified."
  • Bucketing: "Generalizes" a sensitive value by replacing it with a range of values. (For example, replacing a specific age with an age range, or temperatures with ranges corresponding to "Hot," "Medium," and "Cold.")
  • Date shifting: Shifts sensitive date values by a random amount of time.
  • Time extraction: Extracts or preserves specified portions of date and time values.

The remainder of this topic covers each different type of de-identification transformation and provides examples of their use.

Transformation methods

The following table lists the transformations that Sensitive Data Protection provides to de-identify sensitive data:

Transformation Object Description Can Reverse1 Referential Integrity2 Input Type
Redaction RedactConfig Redacts a value by removing it. Any
Replacement ReplaceValueConfig Replaces each input value with a given value. Any
Replace with dictionary ReplaceDictionaryConfig Replaces an input value with a value that is randomly selected from a word list. Any
Replace with infoType ReplaceWithInfoTypeConfig Replaces an input value with the name of its infoType. Any
Mask with character CharacterMaskConfig Masks a string either fully or partially by replacing a given number of characters with a specified fixed character. Any
Pseudonymization by replacing input value with cryptographic hash CryptoHashConfig Replaces input values with a 32-byte hexadecimal string generated using a given data encryption key. See pseudonymization conceptual documentation to learn more. Strings or integers
Pseudonymization by replacing with cryptographic format preserving token CryptoReplaceFfxFpeConfig Replaces an input value with a token, or surrogate value, of the same length using format-preserving encryption (FPE) with the FFX mode of operation. This allows the output to be used in systems that have format validation on length. This is useful for legacy systems where string length must be maintained. Important: For input that varies in length or has length greater than 32 bytes, use CryptoDeterministicConfig. To remain secure the following limits are recommended by the National Institute of Standards and Technology:
  • radix^max_size <= 2^128.
  • radix^min_len >= 100
Strings or integers with a limited number of characters and of uniform length. The alphabet must be made up of at least 2 characters and contain no more than 95.
Pseudonymization by replacing with cryptographic token CryptoDeterministicConfig Replaces an input value with a token, or surrogate value, of the same length using AES in Synthetic Initialization Vector mode (AES-SIV). This transformation method, unlike format-preserving tokenization, has no limitation on supported string character sets, generates identical tokens for each instance of an identical input value, and uses surrogates to enable re-identification given the original encryption key. Any
Bucket values based on fixed size ranges FixedSizeBucketingConfig Masks input values by replacing them with buckets, or ranges within which the input value falls. Any
Bucket values based on custom size ranges BucketingConfig Buckets input values based on user-configurable ranges and replacement values. Any
Date Shifting DateShiftConfig Shifts dates by a random number of days, with the option to be consistent for the same context.
Preserves sequence and duration
Dates/Times
Extract time data TimePartConfig Extracts or preserves a portion of Date, Timestamp, and TimeOfDay values. Dates/Times

Footnotes

1 Reversible transformations can be reversed to re-identify the sensitive data using the content.reidentify method.
2 Referential integrity allows for records to maintain their relationship to one another while still de-identifying the data. For example, given the same crypto key and context, the data will be replaced with the same obfuscuted form each time it is transformed allowing for connections between records to be preserved.

Redaction

If you want to simply remove sensitive data from your input content, Sensitive Data Protection supports a redaction transformation (RedactConfig in the DLP API).

For example, suppose you want to perform a simple redaction of all EMAIL_ADDRESS infoTypes, and the following string is sent to Sensitive Data Protection:

My name is Alicia Abernathy, and my email address is aabernathy@example.com.

The returned string will be the following:

My name is Alicia Abernathy, and my email address is .

The following JSON example and code in several languages shows how to form the API request and what the DLP API returns.

C#

To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.

To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.


using System;
using System.Collections.Generic;
using Google.Api.Gax.ResourceNames;
using Google.Cloud.Dlp.V2;

public class DeidentifyDataUsingRedactWithMatchedInputValues
{
    public static DeidentifyContentResponse Deidentify(
        string projectId,
        string text,
        IEnumerable<InfoType> infoTypes = null)
    {
        // Instantiate the client.
        var dlp = DlpServiceClient.Create();

        // Construct inspect config.
        var inspectConfig = new InspectConfig
        {
            InfoTypes = { infoTypes ?? new InfoType[] { new InfoType { Name = "EMAIL_ADDRESS" } } },
        };

        // Construct redact config.
        var redactConfig = new RedactConfig();

        // Construct deidentify config using redact config.
        var deidentifyConfig = new DeidentifyConfig
        {
            InfoTypeTransformations = new InfoTypeTransformations
            {
                Transformations =
                {
                    new InfoTypeTransformations.Types.InfoTypeTransformation
                    {
                        PrimitiveTransformation = new PrimitiveTransformation
                        {
                            RedactConfig = redactConfig
                        }
                    }
                }
            }
        };

        // Construct a request.
        var request = new DeidentifyContentRequest
        {
            ParentAsLocationName = new LocationName(projectId, "global"),
            DeidentifyConfig = deidentifyConfig,
            InspectConfig = inspectConfig,
            Item = new ContentItem { Value = text }
        };

        // Call the API.
        var response = dlp.DeidentifyContent(request);

        // Check the deidentified content.
        Console.WriteLine($"Deidentified content: {response.Item.Value}");
        return response;
    }
}

Go

To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.

To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

import (
	"context"
	"fmt"
	"io"

	dlp "cloud.google.com/go/dlp/apiv2"
	"cloud.google.com/go/dlp/apiv2/dlppb"
)

// deidentifyWithRedact de-identify the data by redacting with matched input values
func deidentifyWithRedact(w io.Writer, projectID, inputStr string, infoTypeNames []string) error {
	// projectID := "my-project-id"
	// inputStr := "My name is Alicia Abernathy, and my email address is aabernathy@example.com."
	// infoTypeNames := []string{"EMAIL_ADDRESS"}

	ctx := context.Background()

	// Initialize a client once and reuse it to send multiple requests. Clients
	// are safe to use across goroutines. When the client is no longer needed,
	// call the Close method to cleanup its resources.
	client, err := dlp.NewClient(ctx)
	if err != nil {
		return fmt.Errorf("dlp.NewClient: %w", err)
	}

	// Closing the client safely cleans up background resources.
	defer client.