Summarization on XSum (PRR)
0.617PRRTAD
Evaluation Results
| Method | Links | |
|---|---|---|
| TADBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.617 | |
| LookBackLensBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.593 | |
| TADBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.55 | |
| LookBackLensBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.528 | |
| SheepsBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.523 | |
| SheepsBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.51 | |
| SATRMDBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.506 | |
| SATRMDBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.494 | |
| TADBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.46 | |
| MINDBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.437 | |
| LookBackLensBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.436 | |
| MINDBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.428 | |
| CCPBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.399 | |
| SAPLMABackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.395 | |
| SheepsBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.361 | |
| PerplexityBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.353 | |
| Mean Token EntropyBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.342 | |
| CCPBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.339 | |
| SATRMDBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.338 | |
| PerplexityBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.325 | |
| MSPBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.321 | |
| Mean Token EntropyBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.312 | |
| FocusBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.31 | |
| MSPBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.303 | |
| FactoscopeBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.292 | |
| SAPLMABackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.288 | |
| FocusBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.266 | |
| Simple FocusBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.254 | |
| Simple FocusBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.248 | |
| SAPLMABackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.245 | |
| CCPBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.24 | |
| PerplexityBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.237 | |
| FactoscopeBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.234 | |
| Mean Token EntropyBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.233 | |
| LUQBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.228 | |
| MINDBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.22 | |
| FocusBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.209 | |
| LUQBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.197 | |
| FactoscopeBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.185 | |
| Semantic DensityBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.172 | |
| Semantic DensityBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.163 | |
| EVL NLI Score entail.Backbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.139 | |
| DegMat NLI Score entail.Backbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.138 | |
| LUQBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.137 | |
| SARBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.128 | |
| Lexical Similarity Rouge-LBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.122 | |
| Simple FocusBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.109 | |
| SARBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.086 | |
| Semantic DensityBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.08 | |
| MSPBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.077 | |
| EigenScoreBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.077 | |
| Lexical Similarity Rouge-LBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.076 | |
| Lexical Similarity Rouge-LBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.059 | |
| EVL NLI Score entail.Backbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.058 | |
| DegMat NLI Score entail.Backbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.058 | |
| SARBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.052 | |
| EigenScoreBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.033 | |
| EVL NLI Score entail.Backbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.033 | |
| DegMat NLI Score entail.Backbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.031 | |
| EigenScoreBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.024 | |
| Semantic EntropyBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.016 | |
| Semantic EntropyBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.016 | |
| Ecc. NLI Score entail.Backbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.012 | |
| Semantic EntropyBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.011 | |
| Ecc. NLI Score entail.Backbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.004 | |
| Ecc. NLI Score entail.Backbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | -0.047 |