Tag Recognizer guide

The Tag Recognizer model helps you solve key problems in understanding your retail shelf, namely recognizing and parsing the tags (for example, the price tag or other label tags) according to user-defined key-value pair entity extraction schema.

This model can serve as the primary AI building block for analyzing and interpreting product image data in retail stores. For example, you can use this model on shelf images that are captured by local cameras or mobile devices.

Tag Recognizer and Product Recognizer use cases

The Product Recognizer and Tag Recognizer models can serve as the primary AI building blocks for analyzing and interpreting the image data around products and tags image data, which you might have captured from retail stores.

The Product recognizer and Tag recognizer models incorporate several core Google AI modeling and data capabilities to help the retailers and/or technical partners to solve key problems in understanding the product and price-tag image data, including:

  • Recognizing, so understanding what product is in the image.
  • Recognizing and parsing the tag (price-tag, for example, or any other text label tags) according to user defined key-value pair entity extraction schema.

In particular, several differentiating Google AI models are included to support these use-cases problem solving, such as

  • Product thumbnail visual embedding model, which turns a product thumbnail image into a numerical feature space representation.
  • Google OCR model, which extracts all texts visible in the image.
  • Google entity extraction model (that you can customize), which turns the raw texts into the user defined key-value pair named entities.

In addition to these Google AI models, the product recognizer also leverages Google's large database of product information. The product data in this Product database includes the product's GTIN / UPC identity, product brand, title, and cross-language descriptions, product logo, and imagery with various packaging variations. The Product database with the previously mentioned product thumbnail visual embedding model enables the Product recognizer model to be able to recognize many products immediately.

The two major AI features are Product Recognizer and Tag Recognizer models.

Tag Recognizer functionality

This model recognizes all text strings in the tag, and then tries to parse the text into the user-defined key-value pair entity extraction schema, such as product item description or price value. It includes the following differentiating Google AI models:

  • The Google OCR technology, which extracts all visible text in the image.

  • The Google entity extraction model which turns the raw text into the user defined key-value pair named entities. Customize this model using Vertex AI. For example, if you mainly care about the product item description, product price value, or sale price, but nothing else, user can define their tag parsing schema as follows:

    key: item_description   value: string
key: regular_price      value: number
key: sale_price         value: number

Tag Parsing Schema

With the customized entity extraction model training, the detected tag item box will then be recognized and parsed in conformance with the user defined schema, for example as follows:

  item_description:   COLLECTION 18PC GFT BX
  regular_price:      1099
  sale_price:         999

Example Output JSON Object

{
  "imageUri": "gs://test_bucket/test_image.jpg",
  "tagRecognitionAnnotations": [
    {
      "entities": [
        {
          "confidence": 0.99646133,
          "mentionText": "NISSIN TOP RAMEN\n\nBOW CHICKEN\n\n",
          "region": {
            "xMax": 0.4618055,
            "xMin": 0.042725038,
            "yMax": 0.45387268,
            "yMin": 0.18415153
          },
          "type":"description"
        },
        {
          "confidence": 0.95828205,
          "mentionText": "$3.90\n",
          "region": {
            "xMax": 0.24819264,
            "xMin": 0.04185935,
            "yMax": 0.96134734,
            "yMin": 0.80382305
          },
          "type":"unit_price"
        },
        {
          "confidence": 0.60659707,
          "mentionText": "$14.99\n",
          "region": {
            "xMax": 0.9754113,
            "xMin": 0.3654699,
            "yMax": 0.92825794,
            "yMin": 0.40368474
          },
          "type":"price"
        }
      ]
    }
  ]
}

Environment setup

This section describes how to interact with Store Vision AI RESTful API.

API_ENDPOINT=visionai.googleapis.com
PROJECT_ID=your project ID

All the create methods require specifying the to-be-created resource name/ID explicitly. You may use a meaningful string identifier, for example, "product-ABC" or a randomly generated identifier for example, UUID.

To grant a person role the editor access to use Store Vision API, please run the following iam binding command:

gcloud projects add-iam-policy-binding PROJECT_ID --member='user:USER_ACCOUNT' --role='roles/visionai.editor'

To grant a service account the editor access, please use the below command:

gcloud projects add-iam-policy-binding PROJECT_ID --member='serviceAccount:SERVICE_ACCOUNT' --role='roles/visionai.editor'

Learn more about IAM binding.

Tag Recognizer user journey

  1. Perform tag detection model customized training by using Vertex AI / AutoML Vision Object Detection feature.
  2. Perform tag entity parsing model training by using Vertex AI / AutoML Vision Object Detection feature with customized OCR engine.
  3. Create an Endpoint with desired Tag Recognition config.
  4. Perform BatchAnalyze with TagRecognition feature. In the backend, the system will identify tags from each input image, analyze the text on each detected tag to produce a structured parsing output. T ## Tag Detection & Entity Parsing model training

You can train the customized Tag Detection model using our existing Vertex AI / AutoML Vision product's image Object Detection model training feature. Though Vertex AI / AutoML Vision Object Detection model training feature provides a fully managed model training experience, it's still your responsibility to prepare a well sampled image dataset with fully labeled object bounding box annotations as the training dataset to feed into the model training console. Google Cloud provides the Vertex AI Data Labeling Service to let you create the data labeling task. Please follow the following Vertex AI data labeling job link for details: /vertex-ai/docs/datasets/data-labeling-job. Provide clear data labeling instructions to the human raters so that they know how to label the tag detection bounding boxes in the images as the training dataset preparation.

To train the Tag Entity Parsing model, you need to prepare a collection of training data, with images and their associated annotations.

  • The image is the already cropped tag image.
  • In each image, you need to define and provide the key entity field (such as product_title, price, unit_price fields.) they want to detect and recognize / parse, and their associated image bounding box coordinate location in this cropped image view.
  • To support the correct recognition / parsing, we also require you to provide the regular expression syntax to characterize each field. This is needed to assist the tag parsing algorithm's training and inference routine.

recognized tag image

Tag Entity Parsing Training Example

For example, with the previous tag entity parsing training example, you can provide one line of annotation info in the training data annotation CSV file as follows:

"image_filepath", "product_title", "(x0, y0, x1, y1)", "", "price", "(x0, y0, x1, y1)", "\\$\\d+\\.\\d{2}", "unit_price", "(x0, y0, x1, y1)", "\\$\\d+\\.\\d{2}"

From the previous example:

  • The "product_title" field has its corresponding box image coordinate "(x0, y0, x1, y1)" and the regular expression constraint for this field is none "".
  • The "price" field has its corresponding box image coordinate "(x0, y0, x1, y1)" and the regular expression constraint for this field is "\$\d+\.\d{2}", which is indicating we are looking to recognize and parse this field with the $ sign at the beginning of text entry, and a few number digits before the "." and two digits after the ".".
  • The "unit_price" field has the same annotation syntax as the "price" field, for example, the box image coordinates "(x0, y0, x1, y1)" and the regular expression constraint for this field is "\$\d+\.\d{2}", which is indicating we are looking to recognize and parse this field with the $ sign at the beginning of text entry, and a few number digits before the "." and two digits after the ".".

Hence, a proper price-tag parsing / entity detection model training data will have a collection of price-tag images, with the annotation in a CSV file with each CSV row entry just like the entry in the previous example.

"image_filepath", "product_title", "(x0, y0, x1, y1)", "", "price", "(x0, y0, x1, y1)", "\\$\\d+\\.\\d{2}", "unit_price", "(x0, y0, x1, y1)", "\\$\\d+\\.\\d{2}"

"image_filepath", "product_title", "(x0, y0, x1, y1)", "", "price", "(x0, y0, x1, y1)", "\\$\\d+\\.\\d{2}", "unit_price", "(x0, y0, x1, y1)", "\\$\\d+\\.\\d{2}"

"image_filepath", "product_title", "(x0, y0, x1, y1)", "", "price", "(x0, y0, x1, y1)", "\\$\\d+\\.\\d{2}", "unit_price", "(x0, y0, x1, y1)", "\\$\\d+\\.\\d{2}"

[...]

You can train the customized Tag Entity Parsing model using our existing Vertex AI /AutoML Vision product's Image Object Detection model training feature, plus the Google OCR engine customization.

As of 07/2022, customized Tag Detection and Tag Entity Parsing model training and deployment to Store Vision AI's BatchAnalyze API is not yet fully supported as an integrated console experience, you're still able to take advantage of this customized Tag Detection and Tag Entity Parsing model training (using Agent Platform Vision's Object Detection feature) and serving them in Store Vision AI's BatchAnalyze API by taking several manual self-serving steps.

API usage - batch analyze inference

Create Endpoint
  • ENDPOINT_ID=Your endpoint id