The BLEU translation quality metric

BLEU (Bilingual Evaluation Understudy) is a metric for evaluating machine-translated text. A BLEU score is a number between zero and one that measures the similarity of the machine-translated text to a set of high-quality reference translations:

  • A score of 0 means that the machine-translated output has no overlap with the reference translation, which indicates a low quality of translation.
  • A score of 1 means there is perfect overlap with the reference translations, which indicates a high quality of translation.

BLEU limitations

  • BLEU is a Corpus-based Metric. The BLEU metric performs poorly when used to evaluate individual sentences. Single sentences can get very low BLEU scores, even when they capture most of the meaning. Because n-gram statistics for individual sentences are less meaningful, BLEU is by design a corpus-based metric, where statistics are accumulated over an entire corpus when computing the score. The BLEU metric can't be factorized for individual sentences.

  • BLEU makes no distinction between content and function words. The BLEU metric doesn't distinguish content from function words. A dropped function word like a gets the same penalty as if the name NASA were erroneously replaced with ESA.

  • BLEU is not good at capturing sentence meaning and grammaticality. Dropping a single word like not can change sentence polarity. Also, taking only n-grams into account with n≤4 ignores long-range dependencies, so BLEU often imposes only a small penalty for ungrammatical sentences.

  • BLEU relies on normalization and tokenization. Prior to computing the BLEU score, the reference and candidate translations are normalized and tokenized. The choice of steps in those processes significantly affect the final BLEU score.

How to interpret BLEU scores

The following is a rough guideline suggesting how to interpret BLEU scores that are expressed as percentages, not decimals:

BLEU % Score Interpretation
< 10 Almost useless
10 - 19 Hard to get the gist
20 - 29 The gist is clear, but has significant grammatical errors
30 - 40 Understandable to good translations
40 - 50 High quality translations
50 - 60 Very high quality, adequate, and fluent translations
> 60 Quality often better than human

The following color gradient can be used as a general scale interpretation of the BLEU score:

General interpretability of scale

Mathematical details

Mathematically, the BLEU score is defined as:

$$ \text{BLEU} = \underbrace{\vphantom{\prod_i^4}\min\Big(1, \exp\big(1-\frac{\text{reference-length}} {\text{output-length}}\big)\Big)}_{\text{brevity penalty}} \underbrace{\Big(\prod_{i=1}^{4} precision_i\Big)^{1/4}}_{\text{n-gram overlap}} $$

with

\[ precision_i = \dfrac{\sum_{\text{snt}\in\text{Cand-Corpus}}\sum_{i\in\text{snt}}\min(m^i_{cand}, m^i_{ref})} {w_t^i = \sum_{\text{snt'}\in\text{Cand-Corpus}}\sum_{i'\in\text{snt'}} m^{i'}_{cand}} \]

where

  • \(m_{cand}^i\hphantom{xi}\) is the count of i-gram in candidate matching the reference translation.
  • \(m_{ref}^i\hphantom{xxx}\) is the count of i-gram in the reference translation.
  • \(w_t^i\hphantom{m_{max}}\) is the total number of i-grams in candidate translation.

The formula consists of two parts: the brevity penalty and the n-gram overlap.

  • Brevity penalty. The brevity penalty penalizes generated translations that are too short compared to the closest reference length with an exponential decay. The brevity penalty compensates for the fact that the BLEU score has no recall term.

  • N-Gram overlap. The n-gram overlap counts how many unigrams, bigrams, trigrams, and four-grams (i=1,...,4) match their n-gram counterpart in the reference translations. This term acts as a precision metric. Unigrams account for adequacy while longer n-grams account for fluency of the translation. To avoid overcounting, the n-gram counts are clipped to the maximal n-gram count occurring in the reference (\(m_{ref}^n\)).

Example: Calculating \(precision_1\)

Consider this reference sentence and candidate translation:

Reference: the cat is on the mat
Candidate: the the the cat mat

The first step is to count the occurrences of each unigram in the reference and the candidate. Note that the BLEU metric is case-sensitive.

Unigram \(m_{cand}^i\hphantom{xi}\) \(m_{ref}^i\hphantom{xxx}\) \(\min(m^i_{cand}, m^i_{ref})\)
the 3 2 2