Modifying infoType detectors to refine scan results

Sensitive Data Protection's built-in infoType detectors are effective at finding common types of sensitive data. Custom infoType detectors enable you to fully customize your own sensitive data detector. Inspection rules help refine the scan results that the Sensitive Data Protection returns by modifying the detection mechanism of a given infoType detector.

If you want to exclude or include more values from the results that are returned by a built-in infoType detector, you can create a new custom infoType from scratch and define all the criteria that Sensitive Data Protection should look for. Alternatively, you can refine the findings that Sensitive Data Protection's built-in or custom detectors return according to criteria that you specify. You can do this by adding inspection rules that can help reduce noise, increase precision and recall, or adjust likelihood certainty of scan findings.

This topic discusses how to use the two types of inspection rules to either exclude certain findings or add additional findings, all according to custom criteria that you specify. Presented in this topic are several scenarios in which you might want to alter an existing infoType detector.

The following types of inspection rules are available:

• Exclusion rules, which help exclude false or unwanted findings.
• Hotword rules, which help detect additional findings in text content.
• Adjustment rules, which help adjust the likelihood of findings based on the context in which they appear.

Target and context infoTypes

This document uses the following terms to refer to infoTypes in a ruleset.

• Target infoType: an infoType that Sensitive Data Protection excludes or adjusts when the conditions defined in the ruleset are met.
• Context infoType: an infoType that, if detected, provides context about a target infoType. Sensitive Data Protection uses context infoTypes to determine whether it must exclude or adjust a target infoType.

Rule ordering and chaining

Sensitive Data Protection applies the rules in the order you specify them in the ruleset. Therefore, the order of your rules can affect the results of the Sensitive Data Protection operation. For an example, see Boost the likelihood of person name in a health document and exclude the health document in this document.

Exclusion rules

Exclusion rules are useful in situations like the following:

• You want to exclude duplicate scan matches in results that are caused by overlapping infoType detectors. For example, you're scanning for email addresses and phone numbers, but you are receiving two hits for email addresses with phone numbers in them, such as "206-555-0764@example.org."
• You're experiencing noise in your scan results. For example, you're seeing the same dummy email address (such as example@example.com") or domain (such as "example.com") returned an inordinate number of times by a scan for legitimate email addresses.
• You have a list of terms, phrases, or combination of characters that you want to exclude from results.
• You want to exclude an entire column of data from results.
• You want to exclude findings that are near a string that matches a regular expression.
• You want to exclude findings in images based on their spatial relationship with other detected findings in the image.

Exclusion rules API overview

Sensitive Data Protection defines an exclusion rule in the ExclusionRule object. Within ExclusionRule you specify one of the following:

• A Dictionary object, which contains a list of strings to exclude from the results.
• A Regex object, which defines a regular expression pattern. Strings that match the pattern are excluded from the results.
• An ExcludeInfoTypes object, which contains an array of infoType detectors. If a finding is matched by any of the infoType detectors listed here, the finding is excluded from the results.

• An ExcludeByHotword object, which contains the following:

  • A regular expression that defines the hotword.
  • A proximity value that defines how near the hotword must be to the finding.

  If the finding is within the set proximity, that finding is excluded from the results. For tables, this exclusion rule type lets you exclude an entire column of data from the results.

• An ExcludeByImageFindings object, which contains the following:

  • A list of context infoTypes, which are used to determine whether to exclude a target infoType finding.

  • An ImageContainmentType object that specifies the required spatial relationship between the bounding boxes of the target and context findings. If this requirement isn't met, Sensitive Data Protection doesn't exclude the target infoType finding.

  Sensitive Data Protection excludes a target infoType finding if the bounding box of a context infoType has the specified relationship with the target infoType finding. When using ExcludeByImageFindings, you must set the matchingType field to MATCHING_TYPE_RULE_SPECIFIC.

Exclusion rule example scenarios

Each of the following JSON snippets illustrates how to configure Sensitive Data Protection for the given scenario.

Omit specific email address from EMAIL_ADDRESS detector scan

The following JSON snippet and code in several languages illustrates how to indicate to Sensitive Data Protection using an InspectConfig that it should avoid matching on "example@example.com" in a scan that uses the infoType detector EMAIL_ADDRESS:

using System;
using System.Collections.Generic;
using System.Linq;
using Google.Api.Gax.ResourceNames;
using Google.Cloud.Dlp.V2;

public class InspectStringWithExclusionDict
{
    public static InspectContentResponse Inspect(string projectId, string textToInspect, List<String> excludedMatchList)
    {
        var dlp = DlpServiceClient.Create();

        var byteContentItem = new ByteContentItem
        {
            Type = ByteContentItem.Types.BytesType.TextUtf8,
            Data = Google.Protobuf.ByteString.CopyFromUtf8(textToInspect)
        };

        var contentItem = new ContentItem { ByteItem = byteContentItem };

        var infoTypes = new string[] { "PHONE_NUMBER", "EMAIL_ADDRESS", "CREDIT_CARD_NUMBER" }.Select(it => new InfoType { Name = it });

        var exclusionRule = new ExclusionRule
        {
            MatchingType = MatchingType.FullMatch,
            Dictionary = new CustomInfoType.Types.Dictionary
            {
                WordList = new CustomInfoType.Types.Dictionary.Types.WordList
                {
                    Words = { excludedMatchList }
                }
            }
        };

        var ruleSet = new InspectionRuleSet
        {
            InfoTypes = { new InfoType { Name = "EMAIL_ADDRESS" } },
            Rules = { new InspectionRule { ExclusionRule = exclusionRule } }
        };

        var config = new InspectConfig
        {
            InfoTypes = { infoTypes },
            IncludeQuote = true,
            RuleSet = { ruleSet }
        };

        var request = new InspectContentRequest
        {
            Parent = new LocationName(projectId, "global").ToString(),
            Item = contentItem,
            InspectConfig = config
        };

        var response = dlp.InspectContent(request);

        Console.WriteLine($"Findings: {response.Result.Findings.Count}");
        foreach (var f in response.Result.Findings)
        {
            Console.WriteLine("\tQuote: " + f.Quote);
            Console.WriteLine("\tInfo type: " + f.InfoType.Name);
            Console.WriteLine("\tLikelihood: " + f.Likelihood);
        }

        return response;
    }
}

import (
	"context"
	"fmt"
	"io"

	dlp "cloud.google.com/go/dlp/apiv2"
	"cloud.google.com/go/dlp/apiv2/dlppb"
)

// inspectStringWithExclusionDictionary inspects a string for sensitive data
// using exclusion dictionary, here the function omits a specific email address
// from an EMAIL_ADDRESS detector scan with an exclusion dictionary
func inspectStringWithExclusionDictionary(w io.Writer, projectID, textToInspect string, excludedMatchList []string) error {
	// projectID := "my-project-id"
	// textToInspect := "Some email addresses: gary@example.com, example@example.com"
	// excludedMatchList := []string{"example@example.com"}

	ctx := context.Background()

	// Initialize a client once and reuse it to send multiple requests. Clients
	// are safe to use across goroutines. When the client is no longer needed,
	// call the Close method to cleanup its resources.
	client, err := dlp.NewClient(ctx)
	if err != nil {
		return err
	}

	// Closing the client safely cleans up background resources.
	defer client.Close()

	// Specify the type and content to be inspected.
	contentItem := &dlppb.ContentItem{
		DataItem: &dlppb.ContentItem_ByteItem{
			ByteItem: &dlppb.ByteContentItem{
				Type: dlppb.ByteContentItem_TEXT_UTF8,
				Data: []byte(textToInspect),
			},
		},
	}

	// Specify the type of info the inspection will look for.
	// See https://cloud.google.com/dlp/docs/infotypes-reference for complete list of info types.
	infoTypes := []*dlppb.InfoType{
		{Name: "PHONE_NUMBER"},
		{Name: "EMAIL_ADDRESS"},
		{Name: "CREDIT_CARD_NUMBER"},
	}

	// Exclude matches from the specified excludedMatchList.
	exclusionRule := &dlppb.ExclusionRule{
		Type: &dlppb.ExclusionRule_Dictionary{
			Dictionary: &dlppb.CustomInfoType_Dictionary{
				Source: &dlppb.CustomInfoType_Dictionary_WordList_{
					WordList: &dlppb.CustomInfoType_Dictionary_WordList{
						Words: excludedMatchList,
					},
				},
			},
		},
		MatchingType: dlppb.MatchingType_MATCHING_TYPE_FULL_MATCH,
	}

	// Construct a ruleset that applies the exclusion rule to the EMAIL_ADDRESSES infoType.
	ruleSet := &dlppb.InspectionRuleSet{
		InfoTypes: []*dlppb.InfoType{
			{Name: "EMAIL_ADDRESS"},
		},
		Rules: []*dlppb.InspectionRule{
			{
				Type: &dlppb.InspectionRule_ExclusionRule{
					ExclusionRule: exclusionRule,
				},
			},
		},
	}

	// Construct the Inspect request to be sent by the client.
	req := &dlppb.InspectContentRequest{
		Parent