Long Document Summarization on BookSum (test)
43.19ROUGE 1Wu et al.
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Wu et al.# Params.=175,000M2023.06 | 43.19 | 10.63 | 17.1 | — | — | — | — | — | |
| CachEDBackbone=BART-large, Parameters=406M2025.01 | 42.8 | 10.5 | 18.8 | 54.4 | — | — | — | — | |
| LED/extractive-abs.# Params.=243M2023.06 | 42.13 | 10.53 | 16.75 | — | — | — | — | — | |
| Kryscinski et al.# Params.=737M2023.06 | 39.87 | 8.01 | 13.99 | — | — | — | — | — | |
| UnlimiformerBase model=PRIMERA, Training method=+early stop w/ Unlimiformer2023.05 | 39.5 | 7.3 | 15.8 | — | — | — | 22.2 | — | |
| UnlimiformerBase model=PRIMERA, Training method=random-encoded training2023.05 | 39.5 | 7.1 | 15.9 | — | — | — | 19.7 | — | |
| CachEDBackbone=BART-base, Parameters=139M2025.01 | 39.4 | 9.2 | 17 | 53.6 | — | — | — | — | |
| SLEDBackbone=BART-large, Parameters=406M2025.01 | 38.9 | 7.5 | 15.8 | 52.4 | — | — | — | — | |
| Standard finetuningBase model=PRIMERA, Training method=Standard finetuning2023.05 | 38.6 | 7.2 | 15.6 | — | — | — | 11.6 | — | |
| UnlimiformerBase model=PRIMERA, Training method=+test Unlimiformer2023.05 | 38.3 | 7.5 | 15.9 | — | — | — | 18.9 | — | |
| UnlimiformerBackbone=PRIMERA, Parameters=447M2025.01 | 38.2 | 7.1 | 16 | — | — | — | — | — | |
| UnlimiformerBase model=PRIMERA, Training method=alternating training2023.05 | 38.2 | 7.1 | 16 | — | — | — | 23.4 | — | |
| UnlimiformerBase model=PRIMERA, Training method=retrieval training2023.05 | 37.9 | 8.2 | 16.3 | — | — | — | 25.5 | — | |
| UnlimiformerBase model=BARTbase, Training method=random-encoded training2023.05 | 37.3 | 6.7 | 15.2 | — | — | — | 20.8 | — | |
| UnlimiformerBase model=BARTbase, Training method=retrieval training2023.05 | 36.8 | 8.3 | 15.7 | — | — | — | 20.3 | — | |
| UnlimiformerBackbone=BART-base, Parameters=139M2025.01 | 36.7 | 7.3 | 15.5 | 51.5 | — | — | — | — | |
| UnlimiformerBase model=BARTbase, Training method=alternating training2023.05 | 36.7 | 7.3 | 15.5 | — | — | — | 20.3 | — | |
| Standard finetuningBase model=BARTbase, Training method=Standard finetuning2023.05 | 36.4 | 7.6 | 15.3 | — | — | — | 10 | — | |
| Memorizing TransformersBase model=BARTbase, Training method=Memorizing Transformers2023.05 | 35.6 | 6.4 | 14.6 | — | — | — | 10.1 | — | |
| UnlimiformerBase model=BARTbase, Training method=+test Unlimiformer2023.05 | 35.5 | 7.7 | 15.4 | — | — | — | 21.9 | — | |
| UnlimiformerBase model=BARTbase, Training method=+early stop w/ Unlimiformer2023.05 | 35.5 | 7.7 | 15.4 | — | — | — | 21.9 | — | |
| HierarchicalBase model=BARTbase, Training method=Hierarchical (Kryściński et al., 2021)2023.05 | 30 | 6 | 11 | — | — | — | — | — | |
| CachEDBackbone=T5-large, Parameters=770M2025.01 | 29.5 | 5.6 | 15.9 | 49.8 | — | — | — | — | |
| SFTBackbone=LED-base, Parameters=162M2025.01 | 26.2 | 3.8 | 16.9 | 47.3 | — | — | — | — | |
| SFTBackbone=BART-large, Parameters=406M2025.01 | 24.7 | 5.8 | 14 | 48.3 | — | — | — | — | |
| SFTBackbone=BART-base, Parameters=139M2025.01 | 23.6 | 5 | 13.2 | 49 | — | — | — | — | |
| SFTBackbone=T5-large, Parameters=770M2025.01 | 19.9 | 3 | 11.4 | 47.2 | — | — | — | — | |
| FlashMemModel Backbone=Qwen 2.5 1.5B Instruct, Inference Strategy=FlashMem2026.01 | 13.77 | — | — | — | — | — | — | — | |
| MemGenModel Backbone=Qwen 2.5 1.5B Instruct, Inference Strategy=MemGen2026.01 | 12.86 | — | — | — | — | — | — | — | |
| CoT-SCModel Backbone=Qwen 2.5 1.5B Instruct, Inference Strategy=CoT-SC2026.01 | 11.62 | — | — | — | — | — | — | — | |
| VanillaModel Backbone=Qwen 2.5 1.5B Instruct, Inference Strategy=Vanilla2026.01 | 11.14 | — | — | — | — | — | — | — | |
| SnapKVModel Backbone=Qwen 3 4B Instruct, Inference Strategy=SnapKV2026.01 | 11.12 | — | — | — | — | — | — | — | |
| FlashMemModel Backbone=Qwen 3 4B Instruct, Inference Strategy=FlashMem2026.01 | 10.99 | — | — | — | — | — | — | — | |
| SnapKVModel Backbone=Qwen 2.5 1.5B Instruct, Inference Strategy=SnapKV2026.01 | 10.38 | — | — | — | — | — | — | — | |
| MemGenModel Backbone=Qwen 3 4B Instruct, Inference Strategy=MemGen2026.01 | 10.18 | — | — | — | — | — | — | — | |
| CoT-SCModel Backbone=Qwen 3 4B Instruct, Inference Strategy=CoT-SC2026.01 | 9.35 | — | — | — | — | — | — | — | |
| VanillaModel Backbone=Qwen 3 4B Instruct, Inference Strategy=Vanilla2026.01 | 8.93 | — | — | — | — | — | — | — | |
| CachEDbackbone=BART Large, size=406M, category=Long Context Modeling2025.05 | — | — | — | 54.4 | — | — | — | — | |
| CachED BARTBackbone=BART-large2025.01 | — | — | — | — | 41.33 | 52.9 | — | — | |
| CoAbackbone=Claude 3 Opus, category=Multi-LLM Agent2025.05 | — | — | — | — | — | — | — | 17.47 | |
| GPT4omode=Zero-Shot, category=Long Context Modeling2025.05 | — | — | — | 47.24 | — | — | — | — | |
| Mistral Largemode=Zero-Shot, size=123B, category=Long Context Modeling2025.05 | — | — | — | 46.42 | — | — | — | — | |
| NEXUSSUMbackbone=Mistral Large, size=123B, category=Multi-LLM Agent2025.05 | — | — | — | 70.7 | — | — | — | 18.27 | |
| SLEDBackbone=BART-large2025.01 | — | — | — | — | 36.92 | 37.7 | — | — | |
| SLEDbackbone=BART Large, size=406M, category=Long Context Modeling2025.05 | — | — | — | 52.4 | — | — | — | — | |
| UnlimiformerBackbone=BART-base2025.01 | — | — | — | — | 35 | 39.3 | — | — | |
| Unlimiformerbackbone=BART Base, size=139M, category=Long Context Modeling2025.05 | — | — | — | 51.5 | — | — | — | — |