BERTScore: Evaluating Text Generation with BERT
About
We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.
Key insights
What problem does BERTScore solve?
It evaluates generated text by comparing candidate and reference sentences using contextual embeddings and cosine similarity instead of exact n-gram matching, capturing paraphrases and semantic equivalence that surface-form metrics miss.
How does it differ from prior metrics?
Unlike BLEU or METEOR, it uses contextualized BERT embeddings for soft matching. Evaluated on 363 machine translation and image captioning systems, it showed stronger human correlation and model selection performance.
Where is semantic text evaluation useful?
BERTScore offers a practical validation metric for machine translation and captioning. The paper reports 192.5 sentence pairs per second on a GTX 1080 Ti and stronger robustness to adversarial paraphrases than the compared metrics.
Stated limitations and future work
No single configuration clearly dominates; idf weighting helps inconsistently. BERTScore struggles detecting factual errors like incorrect dates or unit conversions. The authors propose designing task-specific metrics and using it as a differentiable training loss.