Summarization Evaluation on SummEval (Correlation Metrics)
0.902Avg Spearman RhoBradley-Terry
Evaluation results
| Method | Links | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Bradley-Terry2025.05 | 0.902 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.771 | |
| Bayesian2025.05 | 0.888 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.917 | |
| Bootstrap2025.05 | 0.885 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.729 | |
| Simple Average2025.05 | 0.885 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.271 | |
| Single Judge2025.05 | 0.787 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.285 | |
| CoRBase Model=Qwen-2.5, Parameter Size=32B2025.12 | 0.6 | 0.506 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DEBATELLM Backbone=GPT-42024.05 | 0.597 | 0.575 | 0.61 | 0.588 | 0.643 | 0.614 | 0.542 | 0.515 | 0.593 | 0.582 | — | — | — | — | — | — | |
| CoRBase Model=DeepSeek2025.12 | 0.586 | 0.467 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4oPrompting=Few-shot2026.02 | 0.58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoTBase Model=Qwen-2.5, Parameter Size=32B2025.12 | 0.569 | 0.477 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BINEVALBackbone=Claude2026.06 | 0.563 | 0.491 | 0.652 | 0.541 | 0.655 | 0.615 | 0.54 | 0.47 | 0.404 | 0.339 | — | — | — | — | — | — | |
| GPT-4oPrompting=Zero-shot2026.02 | 0.55 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ScoringBase Model=Qwen-2.5, Parameter Size=32B2025.12 | 0.546 | 0.453 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| G-EvalLLM Backbone=GPT-42024.05 | 0.533 | 0.45 | 0.522 | 0.42 | 0.6 | 0.553 | 0.495 | 0.416 | 0.517 | 0.412 | — | — | — | — | — | — | |
| ChatEvalLLM Backbone=GPT-42024.05 | 0.528 | 0.458 | 0.477 | 0.412 | 0.595 | 0.516 | 0.501 | 0.432 | 0.539 | 0.472 | — | — | — | — | — | — | |
| DeepSEEK-R1Base Model=DeepSeek2025.12 | 0.521 | 0.432 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| G-EvalBackbone=GPT-42026.06 | 0.514 | 0.418 | 0.582 | 0.457 | 0.507 | 0.425 | 0.506 | 0.455 | 0.547 | 0.433 | — | — | — | — | — | — | |
| MultiAgentLLM Backbone=GPT-42024.05 | 0.507 | 0.459 | 0.486 | 0.41 | 0.628 | 0.596 | 0.41 | 0.393 | 0.505 | 0.436 | — | — | — | — | — | — | |
| CoTBase Model=DeepSeek2025.12 | 0.493 | 0.401 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UniEval2024.05 | 0.474 | 0.377 | 0.575 | 0.442 | 0.446 | 0.371 | 0.449 | 0.371 | 0.426 | 0.325 | — | — | — | — | — | — | |
| UniEvalBackbone=T52026.06 | 0.474 | 0.377 | 0.575 | 0.442 | 0.446 | 0.371 | 0.449 | 0.371 | 0.426 | 0.325 | — | — | — | — | — | — | |
| DEBATELLM Backbone=GPT-3.52024.05 | 0.472 | 0.447 | 0.565 | 0.521 | 0.516 | 0.492 | 0.397 | 0.386 | 0.41 | 0.389 | — | — | — | — | — | — | |
| RuAE-7BBase Model=Qwen-2.5, Parameter Size=7B2025.12 | 0.466 | 0.404 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QwenPrompting=Zero-shot2026.02 | 0.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoRBase Model=Qwen-2.5, Parameter Size=7B2025.12 | 0.456 | 0.383 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QwenPrompting=Few-shot2026.02 | 0.45 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BINEVALBackbone=gpt-oss2026.06 | 0.447 | 0.399 | 0.523 | 0.448 | 0.585 | 0.548 | 0.252 | 0.235 | 0.428 | 0.366 | — | — | — | — | — | — | |
| G-EvalBackbone=gpt-oss2026.06 | 0.436 | 0.392 | 0.451 | 0.392 | 0.559 | 0.527 | 0.217 | 0.203 | 0.515 | 0.446 | — | — | — | — | — | — | |
| M-2Type=Task-Specific Baseline2025.12 | 0.435 | 0.403 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DEBATELLM Backbone=Gemini Pro2024.05 | 0.421 | 0.356 | 0.467 | 0.365 | 0.521 | 0.501 | 0.311 | 0.302 | 0.386 | 0.256 | — | — | — | — | — | — | |
| ScoringBase Model=Qwen-2.5, Parameter Size=7B2025.12 | 0.401 | 0.335 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MixtralPrompting=Few-shot2026.02 | 0.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoTBase Model=Qwen-2.5, Parameter Size=7B2025.12 | 0.398 | 0.329 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPTScore2024.05 | 0.394 | 0.344 | 0.284 | 0.242 | 0.506 | 0.48 | 0.415 | 0.39 | 0.306 | 0.265 | — | — | — | — | — | — | |
| G-EvalLLM Backbone=GPT-3.52024.05 | 0.387 | 0.32 | 0.522 | 0.406 | 0.396 | 0.331 | 0.277 | 0.267 | 0.354 | 0.277 | — | — | — | — | — | — | |
| BARTScore2024.05 | 0.385 | 0.305 | 0.448 | 0.342 | 0.382 | 0.315 | 0.356 | 0.292 | 0.356 | 0.273 | — | — | — | — | — | — | |
| BARTScore2026.06 | 0.385 | 0.305 | 0.448 | 0.342 | 0.382 | 0.315 | 0.356 | 0.292 | 0.356 | 0.273 | — | — | — | — | — | — | |
| MixtralPrompting=Zero-shot2026.02 | 0.38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| M-1Type=Task-Specific Baseline2025.12 | 0.364 | 0.323 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MultiAgentLLM Backbone=GPT-3.52024.05 | 0.35 | 0.309 | 0.407 | 0.348 | 0.354 | 0.325 | 0.241 | 0.223 | 0.398 | 0.341 | — | — | — | — | — | — | |
| LlamaPrompting=Few-shot2026.02 | 0.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| G-EvalLLM Backbone=Gemini Pro2024.05 | 0.29 | 0.25 | 0.35 | 0.279 | 0.403 | 0.368 | 0.158 | 0.148 | 0.248 | 0.206 | — | — | — | — | — | — | |
| GPT-4o-miniPrompting=Few-shot2026.02 | 0.28 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UniEvalBackbone=gpt-oss2026.06 | 0.254 | 0.235 | 0.237 | 0.208 | 0.489 | 0.476 | 0 | 0 | 0.288 | 0.256 | — | — | — | — | — | — | |
| LlamaPrompting=Zero-shot2026.02 | 0.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4o-miniPrompting=Zero-shot2026.02 | 0.24 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERTScore2024.05 | 0.225 | 0.175 | 0.284 | 0.211 | 0.11 | 0.09 | 0.193 | 0.158 | 0.312 | 0.243 | — | — | — | — | — | — | |
| BERTScore2026.06 | 0.225 | 0.175 | 0.284 | 0.211 | 0.11 | 0.09 | 0.193 | 0.158 | 0.312 | 0.243 | — | — | — | — | — | — | |
| ROUGE-12026.06 | 0.192 | 0.15 | 0.167 | 0.126 | 0.16 | 0.13 | 0.115 | 0.094 | 0.326 | 0.252 | — | — | — | — | — | — | |
| MoverScore2024.05 | 0.191 | 0.148 | 0.159 | 0.118 | 0.157 | 0.127 | 0.129 | 0.105 | 0.318 | 0.244 | — | — | — | — | — | — | |
| MoverScore2026.06 | 0.191 | 0.148 | 0.159 | 0.118 | 0.157 | 0.127 | 0.129 | 0.105 | 0.318 | 0.244 | — | — | — | — | — | — | |
| QWQ-32BBase Model=Qwen-2.5, Parameter Size=32B2025.12 | 0.19 | 0.151 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFT-7BBase Model=Qwen-2.5, Parameter Size=7B2025.12 | 0.168 | 0.132 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ROUGE-L2024.05 | 0.165 | 0.128 | 0.128 | 0.099 | 0.115 | 0.092 | 0.105 | 0.084 | 0.311 | 0.237 | — | — | — | — | — | — | |
| ScoringBase Model=DeepSeek2025.12 | 0.009 | 0.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gemini 2.5 FlashTemperature=0, API=OpenRouter2026.01 | — | — | 0.89 | — | 0.55 | — | 0.89 | — | 0.91 | — | 0.25 | 0.07 | 0.21 | 0.09 | — | — | |
| GPT-4oTemperature=0, API=OpenRouter2026.01 | — | — | 0.94 | — | 0.91 | — | 0.9 | — | 0.92 | — | 0.05 | 0.05 | 0.06 | 0.29 | — | — | |
| GPT-4o-miniTemperature=0, API=OpenRouter2026.01 | — | — | 0.92 | — | 0.88 | — | 0.89 | — | 0.93 | — | 0.04 | 0.92 | 0.6 | 0.06 | — | — | |
| Jury-on-DemandJury Configuration=Dynamic2025.12 | — | 0.72 | — | — | — | — | — | — | — | — | — | — | — | — | 0.05 | — | |
| LLaMA-4-MaverickTemperature=0, API=OpenRouter2026.01 | — | — | 0.84 | — | 0.63 | — | 0.81 | — | 0.83 | — | 0.36 | 0.25 | 0.13 | 0.18 | — | — | |
| LLaMA-4-ScoutTemperature=0, API=OpenRouter2026.01 | — | — | 0.88 | — | 0.78 | — | 0.92 | — | 0.92 | — | 0.17 | 0.07 | 0.15 | 0.15 | — | — | |
| Qwen3-235B-A22B-instructTemperature=0, API=OpenRouter2026.01 | — | — | 0.92 | — | 0.66 | — | 0.91 | — | 0.89 | — | 0.09 | 0.08 | 0.09 | 0.16 | — | — | |
| Qwen3-30B-A3B-instructTemperature=0, API=OpenRouter2026.01 | — | — | 0.87 | — | 0.6 | — | 0.9 | — | 0.86 | — | 0.17 | 0.15 | 0.15 | 0.22 | — | — | |
| Static JuryJury Configuration=Average-All2025.12 | — | 0.6 | — | — | — | — | — | — | — | — | — | — | — | — | 0.06 | — | |
| Static JuryJury Configuration=Average-TopK2025.12 | — | 0.67 | — | — | — | — | — | — | — | — | — | — | — | — | 0.08 | — | |
| Static JuryJury Configuration=Weighted-Regression2025.12 | — | 0.69 | — | — | — | — | — | — | — | — | — | — | — | — | 0.04 | — | |
| Static JuryJury Configuration=Weighted-Tau2025.12 | — | 0.61 | — | — | — | — | — | — | — | — | — | — | — | — | 0.05 | — |