Document Summarization on BookSum
46.62ROUGE-1 ScoreExtractive Oracle
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Extractive Oracle2022.03 | 46.62 | 9.17 | 18.31 | — | |
| Top Down TransformerParameters=464M2022.03 | 44.19 | 10.89 | 16.13 | — | |
| GPT-3 RLParameters=175B, Strategy=first subtree RL2022.03 | 43.19 | 10.63 | 17.1 | — | |
| CachED BARTlargeContext Length=128K2025.01 | 42.8 | 10.5 | 18.8 | 54.4 | |
| GPT-3 RLParameters=175B, Strategy=full tree RL2022.03 | 41.51 | 10.46 | 16.88 | — | |
| T5Strategy=Divide-and-conquer2022.03 | 39.46 | 7.69 | 13.77 | — | |
| GPT-3 RLParameters=6B, Strategy=full tree RL2022.03 | 36.79 | 7.22 | 14.84 | — | |
| PegasusStrategy=Divide-and-conquer2022.03 | 35.29 | 6.79 | 12.71 | — | |
| GPT-4oProtocol=Zero-shot (ZS), Context Length=128K2025.01 | 29.97 | 6.44 | 27.92 | 51.62 | |
| BARTStrategy=Divide-and-conquer2022.03 | 29.97 | 6.02 | 10.97 | — | |
| Llama 3.1 8BProtocol=LoRA Fine-Tuning (FT), Context Length=128K2025.01 | 29.19 | 4.17 | 28.06 | 50.12 | |
| MemGenBackbone=Llama 3.1 8B Instruct2026.01 | 15.61 | — | — | — | |
| FlashMemBackbone=Llama 3.1 8B Instruct2026.01 | 14.65 | — | — | — | |
| CoT-SCBackbone=Llama 3.1 8B Instruct2026.01 | 14.62 | — | — | — | |
| VanillaBackbone=Llama 3.1 8B Instruct2026.01 | 13.12 | — | — | — | |
| Llama 3.1 8BProtocol=Zero-shot (ZS), Context Length=128K2025.01 | 12.61 | 2.12 | 11.81 | 41.28 | |
| MemGenBackbone=Llama 3.2 3B Instruct2026.01 | 12.09 | — | — | — | |
| SnapKVBackbone=Llama 3.1 8B Instruct2026.01 | 11.67 | — | — | — | |
| FlashMemBackbone=Llama 3.2 3B Instruct2026.01 | 11.04 | — | — | — | |
| SnapKVBackbone=Llama 3.2 3B Instruct2026.01 | 10.25 | — | — | — | |
| CoT-SCBackbone=Llama 3.2 3B Instruct2026.01 | 9.29 | — | — | — | |
| VanillaBackbone=Llama 3.2 3B Instruct2026.01 | 9.03 | — | — | — |