BLEU (Bilingual Evaluation Understudy) is a classic metric for evaluating machine translation, introduced by IBM researchers in 2002. It compares a candidate translation against one or more reference translations and computes a score from 0 to 100 based on how many n-grams (consecutive words) overlap. Higher means more similar to the reference.
For two decades, BLEU was the dominant translation metric. It is fast, deterministic and easy to optimise against. Every WMT translation competition relied on it, and improvements in BLEU drove the rise of statistical and then neural machine translation systems through the 2010s.
BLEU is now largely deprecated for serious evaluation. The reasons:
- N-gram overlap is a poor proxy for meaning — a paraphrase that conveys the meaning perfectly can have low BLEU; a word salad that copies n-grams from the reference can have high BLEU.
- It punishes legitimate variation — the same sentence can be translated five equally good ways.
- It does not capture fluency or grammaticality — only surface-level overlap.
- It does not work well across languages — high-morphology languages get unfairly low scores.
- It is dominated by neural metrics — COMET, BLEURT and learned metrics correlate far better with human judgement.
In 2026 the modern translation evaluation stack looks like this:
- COMET / COMET-22 — the de facto standard, a neural metric trained to predict human ratings.
- BLEURT — Google's learned metric, also strong.
- Direct assessment by humans — still the gold standard for high-stakes evaluation.
- LLM-as-judge — using GPT-5 or Claude Sonnet 4 to rate translations on a rubric; cheap, scalable, and surprisingly well-correlated with human ratings when the prompt is solid.
- BLEU — kept for backwards comparability with old papers, rarely the headline number.
For a US team shipping translation features, the practical advice is to use COMET or LLM-as-judge for evaluation, keep BLEU as a sanity check and never optimise against BLEU directly. The gap between "BLEU went up" and "users perceive translations as better" is wide enough to mislead a roadmap. The same lesson applies to any task where a simple text-overlap metric is being used as a substitute for harder semantic evaluation.