Create a large custom dictionary detector

This topic describes how to create and rebuild large custom dictionaries. It also covers several error scenarios.

When to choose a large custom dictionary over a regular custom dictionary

Regular custom dictionary detectors are sufficient when you have tens of thousands of sensitive words or phrases that you want to scan your content for. If you have more or if your term list changes frequently, consider creating a large custom dictionary, which can support tens of millions of terms.

How large custom dictionaries differ from other custom infoTypes

Large custom dictionaries are different from other custom infoTypes in that each large custom dictionary has two components:

• A list of phrases that you create and define. The list is stored as either a text file within Cloud Storage or a column in a BigQuery table.
• The dictionary files, which Sensitive Data Protection generates and stores in Cloud Storage. Dictionary files are composed of a copy of your term list plus bloom filters, which aid in searching and matching.

Create a large custom dictionary

This section describes how to create, edit, and rebuild a large custom dictionary.

Create a term list

Create a list that contains all the words and phrases that you want the new infoType detector to search for. Do one of the following:

• Place a text file with each word or phrase on its own line into a Cloud Storage bucket.
• Designate one column of a BigQuery table as the container for the words and phrases. Give each entry its own row in the column. You can use an existing BigQuery table, as long as all dictionary words and phrases are in a single column.

It's possible to assemble a term list that is too large for Sensitive Data Protection to process. If you see an error message, see Troubleshooting errors later in this topic.

Create a stored infoType

After you create your term list, use Sensitive Data Protection to create a dictionary:

using System;
using Google.Api.Gax.ResourceNames;
using Google.Cloud.Dlp.V2;

public class CreateStoredInfoTypes
{
    public static StoredInfoType Create(
        string projectId,
        string outputPath,
        string storedInfoTypeId)
    {
        // Instantiate the dlp client.
        var dlp =