Summarization on SamSum (PRR)
-0.113PRRFactoscope
Evaluation Results
| Method | Links | |
|---|---|---|
| FactoscopeBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | -0.113 | |
| FactoscopeBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | -0.032 | |
| EigenScoreBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | -0.01 | |
| MSPBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.012 | |
| CCPBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.025 | |
| Ecc. NLI Score entail.Backbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.032 | |
| Lexical Similarity Rouge-LBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.057 | |
| Ecc. NLI Score entail.Backbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.069 | |
| Semantic EntropyBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.074 | |
| EigenScoreBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.079 | |
| CCPBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.108 | |
| Semantic EntropyBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.114 | |
| Simple FocusBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.116 | |
| Semantic DensityBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.122 | |
| MSPBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.125 | |
| SARBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.129 | |
| MINDBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.133 | |
| FocusBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.144 | |
| EVL NLI Score entail.Backbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.145 | |
| DegMat NLI Score entail.Backbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.145 | |
| LUQBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.17 | |
| Simple FocusBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.174 | |
| Lexical Similarity Rouge-LBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.186 | |
| Semantic DensityBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.198 | |
| LUQBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.207 | |
| SARBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.215 | |
| PerplexityBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.25 | |
| EVL NLI Score entail.Backbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.261 | |
| PerplexityBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.263 | |
| DegMat NLI Score entail.Backbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.263 | |
| MINDBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.274 | |
| Mean Token EntropyBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.278 | |
| Mean Token EntropyBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.28 | |
| SheepsBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.313 | |
| SATRMDBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.322 | |
| FocusBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.323 | |
| SAPLMABackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.326 | |
| SheepsBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.327 | |
| SATRMDBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.33 | |
| LookBackLensBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.386 | |
| Ecc. NLI Score entail.Backbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 0.4 | |
| SAPLMABackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.406 | |
| TADBackbone=Qwen-2.5 7b, Reference Quality Metric=AlignScore2024.08 | 0.416 | |
| TADBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.498 | |
| LookBackLensBackbone=Gemma-2 9b, Reference Quality Metric=AlignScore2024.08 | 0.501 | |
| FactoscopeBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 6.4 | |
| CCPBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 6.5 | |
| EigenScoreBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 7 | |
| Semantic EntropyBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 8.3 | |
| EVL NLI Score entail.Backbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 8.6 | |
| DegMat NLI Score entail.Backbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 8.9 | |
| MSPBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 10.7 | |
| Semantic DensityBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 12.2 | |
| Lexical Similarity Rouge-LBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 13.2 | |
| SARBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 16.6 | |
| LUQBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 21 | |
| Simple FocusBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 29 | |
| FocusBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 32.4 | |
| MINDBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 36.1 | |
| SAPLMABackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 38.2 | |
| PerplexityBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 40.8 | |
| Mean Token EntropyBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 42.4 | |
| LookBackLensBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 44.1 | |
| SheepsBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 46.6 | |
| SATRMDBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 49.5 | |
| TADBackbone=Llama-3.1 8b, Correctness Metric=AlignScore2024.08 | 53.5 |