Content Classification Tutorial

Audience

This tutorial is designed to let you quickly start exploring and developing applications with the Cloud Natural Language API. It is designed for people familiar with basic programming, though even without much programming knowledge, you should be able to follow along. Having walked through this tutorial, you should be able to use the Reference documentation to create your own basic applications.

This tutorial steps through a Natural Language application using Python code. The purpose here is not to explain the Python client libraries, but to explain how to make calls to the Natural Language API. Applications in Java and Node.js are essentially similar. Consult the Natural Language API Samples for samples in other languages (including the sample in this tutorial).

Prerequisites

This tutorial has several prerequisites:

• You've set up a Cloud Natural Language project in the Google Cloud console.
• You've set up your environment using Application Default Credentials in the Google Cloud console.
• You are familiar with Python in the Google Cloud console programming.
• You have set up your Python development environment. It is recommended that you have the latest version of Python, pip, and virtualenv installed on your system. For instructions, see the Python Development Environment Setup Guide for Google Cloud Platform.
• You've installed the Google Cloud Client Library for Python

Overview

This tutorial walks you through a basic Natural Language application, using classifyText requests, which classifies content into categories along with a confidence score, such as:

category: "/Internet & Telecom/Mobile & Wireless/Mobile Apps & Add-Ons"
confidence: 0.6499999761581421

To see the list of all available category labels, see Categories.

In this tutorial, you will create an application to perform the following tasks:

• Classify multiple text files and write the result to an index file.
• Process input query text to find similar text files.
• Process input query category labels to find similar text files.

The tutorial uses content from Wikipedia. You could create a similar application to process news articles, online comments, and so on.

Source Files

You can find the tutorial source code in the Python Client Library Samples on GitHub.

This tutorial uses sample source text from Wikipedia. You can find the sample text files in the resources/texts folder of the GitHub project.

Importing libraries

To use the Cloud Natural Language API, you must to import the language module from the google-cloud-language library. The language.types module contains classes that are required for creating requests. The language.enums module is used to specify the type of the input text. This tutorial classifies plain text content (language.enums.Document.Type.PLAIN_TEXT).

To calculate the similarity between text based on their resulting content classification, this tutorial uses numpy for vector calculations.

import argparse
import json
import os

from google.cloud import language_v1
import numpy

Step 1. Classify content

You can use the Python client library to make a request to the Natural Language API to classify content. The Python client library encapsulates the details for requests to and responses from the Natural Language API.

The classify function in the tutorial calls the Natural Language API classifyText method, by first creating an instance of the LanguageServiceClient class, and then calling the classify_text method of the LanguageServiceClient instance.

The tutorial classify function only classifies text content for this example. You can also classify the content of a web page by passing in the source HTML of the web page as the text and by setting the type parameter to language.enums.Document.Type.HTML.

For more information, see Classifying Content. For details about the structure of requests to the Natural Language API, see the Natural Language Reference.

def classify(text, verbose=True):
    """Classify the input text into categories."""

    language_client = language_v1.LanguageServiceClient()

    document = language_v1.Document(
        content=text, type_=language_v1.Document.Type.PLAIN_TEXT
    )
    response = language_client.classify_text(request={"document": document})
    categories = response.categories

    result = {}

    for category in categories:
        # Turn the categories into a dictionary of the form:
        # {category.name: category.confidence}, so that they can
        # be treated as a sparse vector.
        result[category.name] = category.confidence

    if verbose:
        print(text)
        for category in categories:
            print("=" * 20)
            print("{:<16}: {}".format("category", category.name))
            print("{:<16}: {}".format("confidence", category.confidence))

    return result

The returned result is a dictionary with the category labels as keys, and confidence scores as values, such as:

{
    "/Computers & Electronics": 0.800000011920929,
    "/Internet & Telecom/Mobile & Wireless/Mobile Apps & Add-Ons": 0.6499999761581421
}

The tutorial Python script is organized so that it can be run from the command line for quick experiments. For example you can run:

python classify_text_tutorial.py classify "Google Home enables users to speak voice commands to interact with services through the Home's intelligent personal assistant called Google Assistant. A large number of services, both in-house and third-party, are integrated, allowing users to listen to music, look at videos or photos, or receive news updates entirely by voice. "

Step 2. Index multiple text files

The index function in the tutorial script takes, as input, a directory containing multiple text files, and the path to a file where it stores the indexed output (the default file name is index.json). The index function reads the content of each text file in the input directory, and then passes the text files to the Cloud Natural Language API to be classified into content categories.

def index(path, index_file):
    """Classify each text file in a directory and write
    the results to the index_file.
    """

    result = {}
    for filename in os.listdir(path):
        file_path = os.path.join(path, filename)

        if not os.path.isfile(file_path):
            continue

        try:
            with open(file_path) as f:
                text = f.read()
                categories = classify(text, verbose=False)

                result[filename] = categories
        except Exception:
            print(f"Failed to process {file_path}")

    with open(index_file, "w", encoding="utf-8") as f:
        f.write(json.dumps(result, ensure_ascii=False))

    print(f"Texts indexed in file: {index_file}")
    return result

The results from the Cloud Natural Language API for each file are organized into a single dictionary, serialized as a JSON string, and then written to a file. For example:

{
    "android.txt": {
        "/Computers & Electronics": 0.800000011920929,
        "/Internet & Telecom/Mobile & Wireless/Mobile Apps & Add-Ons": 0.6499999761581421
    },
    "google.txt": {
        "/Internet & Telecom": 0.5799999833106995,
        "/Business & Industrial": 0.5400000214576721
    }
}

To index text files from the command line with the default output filename index.json, run the following command:

python classify_text_tutorial.py index resources/texts

Step 3. Query the index

Query with category labels

Once the index file (default file name = index.json) has been created, we can make queries to the index to retrieve some of the filenames and their confidence scores.

One way to do this is to use a category label as the query, which the tutorial accomplishes with the query_category function. The implementation of the helper functions, such as similarity, can be found in the classify_text_tutorial.py file. In your applications the similarity scoring and ranking should be carefully designed around specific use cases.