Summarization on MENSA (test)
65.73BERTScore F1NEXUSSUM
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| NEXUSSUMbackbone=Mistral Large, size=123B, category=Multi-LLM Agent2025.05 | 65.73 | — | — | — | — | — | — | — | — | |
| CachEDbackbone=BART Large, size=406M, category=Long Context Modeling2025.05 | 64.6 | — | — | — | — | — | — | — | — | |
| HM-SRbackbone=GPT4o-mini, category=Multi-LLM Agent2025.05 | 60.22 | — | — | — | — | — | — | — | — | |
| Unlimiformerbackbone=BART Base, size=139M, category=Long Context Modeling2025.05 | 58.7 | — | — | — | — | — | — | — | — | |
| SLEDbackbone=BART Large, size=406M, category=Long Context Modeling2025.05 | 58.3 | — | — | — | — | — | — | — | — | |
| Summ Nbackbone=LED-Large, size=459M, category=Extractive-to-Abstractive2025.05 | 56.34 | — | — | — | — | — | — | — | — | |
| Mistral Largemode=Zero-Shot, size=123B, category=Long Context Modeling2025.05 | 54.8 | — | — | — | — | — | — | — | — | |
| Select and Summbackbone=LED-Large, size=459M, category=Extractive-to-Abstractive2025.05 | 40.87 | — | — | — | — | — | — | — | — | |
| Agent-as-a-Judgemerging_strategy=hierarchical merging, iteration=1st, backbone=GPT-4o-mini2025.01 | — | 33 | 9.7 | 12.84 | 60.22 | 60.11 | 60.16 | 85.94 | — | |
| Agent-as-a-Judgemerging_strategy=hierarchical merging, iteration=2nd, backbone=GPT-4o-mini2025.01 | — | 33.75 | 9.72 | 13.07 | 60.17 | 60.1 | 60.12 | 88.94 | — | |
| Agent-as-a-Judgemerging_strategy=hierarchical merging, iteration=3rd, backbone=GPT-4o-mini2025.01 | — | 34.26 | 9.74 | 13.46 | 60.24 | 60.21 | 60.22 | 93.31 | — | |
| GPT-4o-minimerging_strategy=hierarchical merging2025.01 | — | 31.79 | 9.69 | 12.68 | 60 | 60.03 | 60.01 | 81.05 | — | |
| LEDmerging_strategy=without merging2025.01 | — | 17.46 | 1.59 | 10.03 | 42.9 | 42.74 | 42.58 | 56.48 | — | |
| LongT5merging_strategy=without merging2025.01 | — | 20.77 | 2.26 | 10.03 | 45.05 | 45.06 | 45.01 | 73.76 | — | |
| TextRankmerging_strategy=without merging2025.01 | — | 34.37 | 4.6 | 12.84 | 46.86 | 49.43 | 48.1 | 59.72 | — |