Create a large custom dictionary detector

This topic describes how to create and rebuild large custom dictionaries. It also covers several error scenarios.

When to choose a large custom dictionary over a regular custom dictionary

Regular custom dictionary detectors are sufficient when you have tens of thousands of sensitive words or phrases that you want to scan your content for. If you have more or if your term list changes frequently, consider creating a large custom dictionary, which can support tens of millions of terms.

How large custom dictionaries differ from other custom infoTypes

Large custom dictionaries are different from other custom infoTypes in that each large custom dictionary has two components:

  • A list of phrases that you create and define. The list is stored as either a text file within Cloud Storage or a column in a BigQuery table.
  • The dictionary files, which Sensitive Data Protection generates and stores in Cloud Storage. Dictionary files are composed of a copy of your term list plus bloom filters, which aid in searching and matching.

Create a large custom dictionary

This section describes how to create, edit, and rebuild a large custom dictionary.

Create a term list

Create a list that contains all the words and phrases that you want the new infoType detector to search for. Do one of the following:

  • Place a text file with each word or phrase on its own line into a Cloud Storage bucket.
  • Designate one column of a BigQuery table as the container for the words and phrases. Give each entry its own row in the column. You can use an existing BigQuery table, as long as all dictionary words and phrases are in a single column.

It's possible to assemble a term list that is too large for Sensitive Data Protection to process. If you see an error message, see Troubleshooting errors later in this topic.

Create a stored infoType

After you create your term list, use Sensitive Data Protection to create a dictionary:

Console

  1. In a Cloud Storage bucket, create a new folder where Sensitive Data Protection will store the generated dictionary.

    Sensitive Data Protection creates folders containing the dictionary files at the location that you specify.

  2. In the Google Cloud console, go to the Create infoType page.

    Go to Create infoType

  3. For Type, select Large custom dictionary.

  4. For InfoType ID, enter an identifier for the stored infoType.

    You will use this identifier when configuring your inspection and de-identification jobs. You can use letters, numbers, hyphens, and underscores in the name.

  5. For InfoType display name, enter a name for your stored infoType.

    You can use spaces and punctuation in the name.

  6. For Description, enter a description of what your stored infoType detects.

  7. For Storage type, select the location of your term list:

    • BigQuery: Enter the project ID, dataset ID, and table ID. In the Field name field, enter the column identifier. You can designate at most one column from the table.
    • Google Cloud Storage: Enter the path to the file.

  8. For Output bucket or folder, enter the Cloud Storage location of the folder that you created in step 1.

  9. Click Create.

A summary of the stored infoType appears. When the dictionary is generated and the new stored infoType is ready to use, the status of the infoType shows Ready.

C#

To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.

To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 


using System;
using Google.Api.Gax.ResourceNames;
using Google.Cloud.Dlp.V2;

public class CreateStoredInfoTypes
{
    public static StoredInfoType Create(
        string projectId,
        string outputPath,
        string storedInfoTypeId)
    {
        // Instantiate the dlp client.
        var dlp = DlpServiceClient.Create();

        // Construct the stored infotype config by specifying the public table and 
        // cloud storage output path.
        var storedInfoTypeConfig = new StoredInfoTypeConfig
        {
            DisplayName = "Github Usernames",
            Description = "Dictionary of Github usernames used in commits.",
            LargeCustomDictionary = new LargeCustomDictionaryConfig
            {
                BigQueryField = new BigQueryField
                {
                    Table = new BigQueryTable
                    {
                        DatasetId = "samples",
                        ProjectId = "bigquery-public-data",
                        TableId = "github_nested"
                    },
                    Field = new FieldId
                    {
                        Name = "actor"
                    }
                },
                OutputPath = new CloudStoragePath
                {
                    Path = outputPath
                }
            },
        };

        // Construct the request.
        var request = new CreateStoredInfoTypeRequest
        {
            ParentAsLocationName = new LocationName(projectId, "global"),
            Config = storedInfoTypeConfig,
            StoredInfoTypeId = storedInfoTypeId
        };

        // Call the API.
        StoredInfoType response = dlp.CreateStoredInfoType(request);

        // Inspect the response.
        Console.WriteLine($"Created the stored infotype at path: {response.Name}");

        return response;
    }
}

Go

To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.

To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 

import (
	"context"
	"fmt"
	"io"

	dlp "cloud.google.com/go/dlp/apiv2"
	"cloud.google.com/go/dlp/apiv2/dlppb"
)

// createStoredInfoType creates a custom stored info type based on your input data.
func createStoredInfoType(w io.Writer, projectID, outputPath string) error {
	// projectId := "my-project-id"
	// outputPath := "gs://" + "your-bucket-name" + "path/to/directory"

	ctx := context.Background()

	// Initialize a client once and reuse it to send multiple requests. Clients
	// are safe to use across goroutines. When the client is no longer needed,
	// call the Close method to cleanup its resources.
	client, err := dlp.NewClient(ctx)
	if err != nil {
		return err
	}

	// Closing the client safely cleans up background resources.
	defer client.Close()

	// Specify the name you want to give the dictionary.
	displayName := "Github Usernames"

	// Specify a description of the dictionary.
	description := "Dictionary of GitHub usernames used in commits"

	// Specify the path to the location in a Cloud Storage
	// bucket to store the created dictionary.
	cloudStoragePath := &dlppb.CloudStoragePath{
		Path: outputPath,
	}

	// Specify your term list is stored in BigQuery.
	bigQueryField := &dlppb.BigQueryField{
		Table: &dlppb.BigQueryTable{
			ProjectId: "bigquery-public-data",
			DatasetId: "samples",
			TableId:   "github_nested",
		},
		Field: &dlppb.FieldId{
			Name: "actor",
		},
	}

	// Specify the configuration of the large custom dictionary.
	largeCustomDictionaryConfig := &dlppb.LargeCustomDictionaryConfig{