Creating and using glossaries (Advanced)

A glossary is a custom dictionary the Cloud Translation API uses to consistently translate the customer's domain-specific terminology. This typically involves specifying how to translate a named entity.

You might use a glossary for the following use cases:

Product names: For example, "Google Home" must translate to "Google Home".
Ambiguous words: For example, the word "bat" can mean a piece of sports equipment or an animal. If you know that you are translating words about sports, you might want to use a glossary to feed the Cloud Translation API the sports translation of "bat", not the translation for the animal.
Borrowed words: For example, "bouillabaisse" in French translates to "bouillabaisse" in English. English borrowed the word "bouillabaisse" from French in the 19th century. An English speaker lacking French cultural context might not know that bouillabaisse is a fish stew dish. Glossaries can override a translation so that "bouillabaisse" in French translates to "fish stew" in English.

Before you begin

Before you can start using the Cloud Translation API, you should complete the Cloud Translation setup page, which includes creating a project, enabling the Cloud Translation API, and setting up authentication. The setup page also provides instructions for installing client libraries for common programming languages, which is optional.

Required permissions

To work with glossaries, your service account requires glossary-specific permissions. You can grant a role to your service account by using one of the pre-defined IAM roles, such as Cloud Translation API Editor (roles/cloudtranslate.editor), or you can create a custom role that grants the necessary permissions. You can view all of the Cloud Translation API permissions in the IAM permissions reference. Cloud Translation permissions begin with cloudtranslate.

For creating glossaries, you also need permissions to read objects in the Cloud Storage bucket where your glossary file is located. You can grant a role to your service account by using one of the pre-defined IAM roles, such as Storage Object Viewer (roles/storage.objectViewer), or you can create a custom role that grants permissions to read objects.

For information on adding an account to a role, see Granting, changing, and revoking access to resources.

Create a glossary

The terms in a glossary can be single tokens (words) or short phrases (usually fewer than five words). The main steps for using a glossary are:

Create a glossary file
Create the glossary resource with our Cloud Translation API
Specify which glossary to use when you request a translation

A project can have multiple glossaries. You can get a list of the available glossaries and can delete glossaries that you no longer need.

Stopwords

Cloud Translation ignores some terms that are included in a glossary; these terms are known as stopwords. When translating stopwords, Cloud Translation ignores any matching glossary entries. For a list of all stopwords, see Glossary stopwords.

Creating a glossary file

Fundamentally, a glossary is a text file in which each line contains corresponding terms in multiple languages. The Cloud Translation API supports both unidirectional glossaries, which specify the desired translation for a single pair of source and target languages, and equivalent term sets, which identify the equivalent terms in multiple languages.

The total number of terms in a glossary input file can't exceed 10.4 million (10,485,760) UTF-8 bytes for all terms in all the languages combined. Any single glossary term must be less than 1024 UTF-8 bytes. Terms longer than 1024 bytes are ignored.

By default, glossary matches are case sensitive. When you apply a glossary, you can ignore case for all entries. If you have a mix of case-sensitive and case-insensitive terms, use the default behavior and, for case-insensitive terms, include both forms in your glossary.

Unidirectional glossaries

The Cloud Translation API accepts TSV, CSV or TMX files.

TSV and CSV

For tab-separated values (TSV) and comma separated values (CSV), each row contains a pair of terms separated by a tab (\t) or comma (,). The first column includes the term in the source language and the second column includes the term in the target language, as shown in the following example:

Unidirectional
glossary

Translation Memory eXchange (TMX)

Translation Memory eXchange (TMX) is a standard XML format for providing source and target translations. The Cloud Translation API supports input files in a format based on TMX version 1.4. This example illustrates the required structure:

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE tmx SYSTEM "tmx14.dtd">
<tmx version="1.4">
  <header segtype="sentence" o-tmf="UTF-8"
  adminlang="en" srclang="en" datatype="PlainText"/>
  <body>
    <tu>
      <tuv xml:lang="en">
        <seg>account</seg>
      </tuv>
      <tuv xml:lang="es">
        <seg>cuenta</seg>
      </tuv>
    </tu>
    <tu>
      <tuv xml:lang="en">
        <seg>directions</seg>
      </tuv>
      <tuv xml:lang="es">
        <seg>indicaciones</seg>
      </tuv>
    </tu>
  </body>
</tmx>

The <header> element of a well-formed TMX file must identify the source language using the srclang attribute, and every <tuv> element must identify the language of the contained text using the xml:lang attribute. You identify the source and target languages using their ISO-639 codes.

All <tu> elements must contain a pair of <tuv> elements with the same source and target languages. If a <tu> element contains more than two <tuv> elements, the Cloud Translation API processes only the first <tuv> matching the source language and the first matching the target language and ignores the rest. If a <tu> element does not have a matching pair of <tuv> elements, the Cloud Translation API skips over the invalid <tu> element.

The Cloud Translation API strips the markup tags from around a <seg> element before processing it. If a <tuv> element contains more than one <seg> element, the Cloud Translation API concatenates their text into a single element with a space between them.

If the file contains XML tags other than those shown above, the Cloud Translation API ignores them.

If the file does not conform to proper XML and TMX format – for example, if it is missing an end tag or a <tmx> element – the Cloud Translation API aborts processing it. The Cloud Translation API also aborts processing if it skips more than 1024 invalid <tu> elements.

Equivalent term sets (CSV)

For equivalent term sets, Cloud Translation API only accepts files in the CSV format. To define equivalent term sets, create a multi-column CSV file in which each row lists a single glossary term in multiple languages.

The first row in the file is a header row identifying the language for each column, using its ISO-639 or BCP-47 language code. You can also include optional columns for part of speech (pos) and a description (description). The algorithm does not currently use pos information, and specific pos values are not validated.

Each subsequent row contains equivalent glossary terms in the languages identified in the header. You can leave columns blank if the term is not available in all languages.

Equivalent terms set

Create a glossary resource

After you have the equivalent glossary terms identified, make the glossary file available to the Cloud Translation API by creating a glossary resource.

Unidirectional glossary

When creating a unidirectional glossary, you must indicate the language pair (language_pair), by specifying the source language (source_language_code), and the target language (target_language_code). The following example uses the REST API and command line, but you can also use the client libraries to create a unidirectional glossary.

REST

When you create a new glossary, you supply a glossary id (a resource name). For example:

projects/my-project/locations/us-central1/glossaries/my-en-to-ru-glossary

where my-project is the PROJECT_NUMBER_OR_ID, and my-en-ru-glossary is the glossary-id provided by you.

Before using any of the request data, make the following replacements:

PROJECT_NUMBER_OR_ID: the numeric or alphanumeric ID of your Google Cloud project
glossary-id: your glossary ID, for example., my_en_ru_glossary
bucket-name: name of bucket where your glossary file is located
glossary-filename: filename of your glossary

HTTP method and URL:

POST https://translation.googleapis.com/v3/projects/PROJECT_NUMBER_OR_ID/locations/us-central1/glossaries

Request JSON body:

{
  "name":"projects/PROJECT_NUMBER_OR_ID/locations/us-central1/glossaries/glossary-id",
  "languagePair": {
    "sourceLanguageCode": "en",
    "targetLanguageCode": "ru"
    },
  "inputConfig": {
    "gcsSource": {
      "inputUri": "gs://bucket-name/glossary-filename"
    }
  }
}

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Note: The following command assumes that you have logged in to the gcloud CLI with your user account by running gcloud init or gcloud auth login , or by using Cloud Shell, which automatically logs you into the gcloud CLI . You can check the currently active account by running gcloud auth list.

Save the request body in a file named request.json, and execute the following command:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "x-goog-user-project: PROJECT_NUMBER_OR_ID" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://translation.googleapis.com/v3/projects/PROJECT_NUMBER_OR_ID/locations/us-central1/glossaries"

PowerShell (Windows)

Note: The following command assumes that you have logged in to the gcloud CLI with your user account by running gcloud init or gcloud auth login . You can check the currently active account by running gcloud auth list.

Save the request body in a file named request.json, and execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "PROJECT_NUMBER_OR_ID" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://translation.googleapis.com/v3/projects/PROJECT_NUMBER_OR_ID/locations/us-central1/glossaries" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "name": "projects/project-number/locations/us-central1/operations/operation-id",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.translation.v3beta1.CreateGlossaryMetadata",
    "name": "projects/project-number/locations/us-central1/glossaries/glossary-id",
    "state": "RUNNING",
    "submitTime": "2019-11-19T19:05:10.650047636Z"
  }
}

Equivalent term sets glossary

Once you have the glossary terms identified in your equivalent term set, make the glossary file available to the Cloud Translation API by creating a glossary resource.

REST

Before using any of the request data, make the following replacements:

PROJECT_NUMBER_OR_ID: the numeric or alphanumeric ID of your Google Cloud project
glossary-id: your glossary ID
bucket-name: name of bucket where your glossary file is located
glossary-filename: filename of your glossary

HTTP method and URL:

POST https://translation.googleapis.com/v3/projects/PROJECT_NUMBER_OR_ID/locations/us-central1/glossaries

Request JSON body:

{
  "name":"projects/PROJECT_NUMBER_OR_ID/locations/us-central1/glossaries/glossary-id",
  "languageCodesSet": {
    "languageCodes": ["en", "en-GB", "ru", "fr", "pt-BR", "pt-PT", "es"]
  },
  "inputConfig": {
    "gcsSource": {
"inputUri": "gs://bucket-name/glossary-file-name"
    }
  }
}

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Save the request body in a file named request.json, and execute the following command:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "x-goog-user-project: PROJECT_NUMBER_OR_ID" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://translation.googleapis.com/v3/projects/PROJECT_NUMBER_OR_ID/locations/us-central1/glossaries"

PowerShell (Windows)

Save the request body in a file named request.json, and execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred"; "x-goog-user-project" = "PROJECT_NUMBER_OR_ID" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://translation.googleapis.com/v3/projects/PROJECT_NUMBER_OR_ID/locations/us-central1/glossaries" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "name": "projects/project-number/locations/us-central1/operations/20191103-09061569945989-5d937985-0000-21ac-816d-f4f5e80782d4",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.translation.v3beta1.CreateGlossaryMetadata",
    "name": "projects/project-number/locations/us-central1/glossaries/glossary-id",
    "state": "RUNNING",
    "submitTime": "2019-11-03T16:06:29.134496675Z"
  }
}

Go

Before trying this sample, follow the Go setup instructions in the Cloud Translation quickstart using client libraries. For more information, see the Cloud Translation Go API reference documentation.

To authenticate to Cloud Translation, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

import (
	"context"
	"fmt"
	"io"

	translate "cloud.google.com/go/translate/apiv3"
	"cloud.google.com/go/translate/apiv3/translatepb"
)

// createGlossary creates a glossary to use for other operations.
func createGlossary(w io.Writer, projectID string, location string, glossaryID string, glossaryInputURI string) error {
	// projectID := "my-project-id"
	// location := "us-central1"
	// glossaryID := "my-glossary-display-name"
	// glossaryInputURI := "gs://cloud-samples-data/translation/glossary.csv"

	ctx := context.Background()
	client, err := translate.NewTranslationClient(ctx)
	if err != nil {
		return fmt.Errorf("NewTranslationClient: %w", err)
	}
	defer client.Close()

	req := &translatepb.CreateGlossaryRequest{
		Parent: fmt.Sprintf("projects/%s/locations/%s", projectID, location),
		Glossary: &translatepb.Glossary{
			Name: fmt.Sprintf("projects/%s/locations/%s/glossaries/%s", projectID, location, glossaryID),
			Languages: &translatepb.Glossary_LanguageCodesSet_{
				LanguageCodesSet: &translatepb.Glossary_LanguageCodesSet{
					LanguageCodes: []string{"en", "ja"},
				},
			},
			InputConfig: &translatepb.GlossaryInputConfig{
				Source: &translatepb.GlossaryInputConfig_GcsSource{
					GcsSource: &translatepb.GcsSource{
						InputUri: glossaryInputURI,
					},
				},
			},
		},
	}

	// The CreateGlossary operation is async.
	op, err := client.CreateGlossary(ctx, req)
	if err != nil {
		return fmt.Errorf("CreateGlossary: %w", err)
	}
	fmt.Fprintf(w, "Processing operation name: %q\n", op.Name())

	resp, err := op.Wait(ctx)
	if err != nil {
		return fmt.Errorf("Wait: %w", err)
	}

	fmt.Fprintf(w, "Created: %v\n", resp.GetName())
	fmt.Fprintf(w, "Input URI: %v\n", resp.InputConfig.GetGcsSource().GetInputUri())

	return nil
}

Java

Before trying this sample, follow the Java setup instructions in the Cloud Translation quickstart using client libraries. For more information, see the Cloud Translation Java API reference documentation.

To authenticate to Cloud Translation, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

import com.google.api.gax.longrunning.OperationFuture;
import com.google.cloud.translate.v3.CreateGlossaryMetadata;
import com.google.cloud.translate.v3.CreateGlossaryRequest;
import com.google.cloud.translate.v3.GcsSource;
import com.google.cloud.translate.v3.Glossary;
import com.google.cloud.translate.v3.GlossaryInputConfig;
import com.google.cloud.translate.v3.GlossaryName;
import com.google.cloud.translate.v3.LocationName;
import com.google.cloud.translate.v3.TranslationServiceClient;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;

public class CreateGlossary {

  public static void createGlossary() throws InterruptedException, ExecutionException, IOException {
    // TODO(developer): Replace these variables before running the sample.
    String projectId = "YOUR-PROJECT-ID";
    String glossaryId = "your-glossary-display-name";
    List<String> languageCodes = new ArrayList<>();
    languageCodes.add("your-language-code");
    String inputUri = "gs://your-gcs-bucket/path/to/input/file.txt";
    createGlossary(projectId, glossaryId, languageCodes, inputUri);
  }

  // Create a equivalent term sets glossary
  // https://cloud.google.com/translate/docs/advanced/glossary#format-glossary
  public static void createGlossary(
      String projectId, String glossaryId, List<String> languageCodes, String inputUri)
      throws IOException, ExecutionException, InterruptedException {

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (TranslationServiceClient client = TranslationServiceClient.create()) {
      // Supported Locations: `global`, [glossary location], or [model location]
      // Glossaries must be hosted in `us-central1`
      // Custom Models must use the same location as your model. (us-central1)
      String location = "us-central1";
      LocationName parent = LocationName.of(projectId, location);
      GlossaryName glossaryName = GlossaryName.of(projectId, location, glossaryId);

      // Supported Languages: https://cloud.google.com/translate/docs/languages
      Glossary.LanguageCodesSet languageCodesSet =