Meta-evaluation on SummEval
0.448Spearman Correlation (COH)BARTSCORE-CNN
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| BARTSCORE-CNNFine-tuning data=CNN/Daily Mail2021.06 | 0.448 | — | — | — | — | 0.382 | 0.356 | 0.356 | 0.499 | — | |
| BARTSCORE-CNN-PARAFine-tuning data=CNN + Paraphrase2021.06 | 0.424 | — | — | — | — | 0.401 | 0.378 | 0.313 | 0.497 | — | |
| BARTSCORE-PROMPTPrompting=true2021.06 | 0.407 | — | — | — | — | 0.378 | 0.338 | 0.368 | 0.518 | — | |
| BARTSCORE2021.06 | 0.322 | — | — | — | — | 0.311 | 0.248 | 0.264 | 0.465 | — | |
| BERTScore2021.06 | 0.284 | — | — | — | — | 0.11 | 0.193 | 0.312 | 0.217 | — | |
| PRISM2021.06 | 0.249 | — | — | — | — | 0.345 | 0.254 | 0.212 | 0.41 | — | |
| ROUGE-22021.06 | 0.184 | — | — | — | — | 0.187 | 0.159 | 0.29 | 0.165 | — | |
| ROUGE-12021.06 | 0.167 | — | — | — | — | 0.16 | 0.115 | 0.326 | 0.194 | — | |
| MoverScore2021.06 | 0.159 | — | — | — | — | 0.157 | 0.129 | 0.318 | 0.2 | — | |
| ROUGE-L2021.06 | 0.128 | — | — | — | — | 0.115 | 0.105 | 0.311 | 0.164 | — | |
| ALIGNSCORE-baseBackbone=ROBERTa (125M)2023.05 | — | — | — | — | — | — | — | — | — | 43.4 | |
| ALIGNSCORE-largeBackbone=ROBERTa (355M)2023.05 | — | — | — | — | — | — | — | — | — | 46.6 | |
| ChatGPTBackbone=GPT3.5-turbo2023.05 | — | — | — | — | — | — | — | — | — | 43.3 | |
| G-EVAL-3.5Backbone=GPT3.5-d032023.05 | — | — | — | — | — | — | — | — | — | 38.6 | |
| G-EVAL-4Backbone=GPT42023.05 | — | — | — | — | — | — | — | — | — | 50.7 | |
| Gemma-2-27BRole=LLM-as-Judge2025.02 | — | 0.527 | 84.2 | 0.533 | 0.49 | — | — | — | — | — | |
| GPT-4 TurboRole=LLM-as-Judge2025.02 | — | 0.741 | 83.9 | 0.625 | 0.513 | — | — | — | — | — | |
| GPT-4oRole=LLM-as-Judge2025.02 | — | 0.58 | 83.5 | 0.562 | 0.522 | — | — | — | — | — | |
| GPT-4o miniRole=LLM-as-Judge2025.02 | — | 0.393 | 81.5 | 0.521 | 0.507 | — | — | — | — | — | |
| GPTScoreBackbone=GPT3.5-d032023.05 | — | — | — | — | — | — | — | — | — | 47.5 | |
| Llama-3.1-70BRole=LLM-as-Judge2025.02 | — | 0.402 | 80.7 | 0.523 | 0.491 | — | — | — | — | — | |
| Phi-4-14BRole=LLM-as-Judge2025.02 | — | 0.68 | 81.5 | 0.564 | 0.532 | — | — | — | — | — | |
| Qwen-2.5-72BRole=LLM-as-Judge2025.02 | — | 0.518 | 84 | 0.577 | 0.49 | — | — | — | — | — |