Long document summarization on ScriptBase (test)
50.2ROUGE 1CachED
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| CachEDBackbone=BART-large, Parameters=406M2025.01 | 50.2 | 14.9 | 20.4 | 64.6 | — | — | |
| SELECT & SUMMbase_model=LED2024.04 | 49.98 | 12.11 | 47.95 | 57.46 | 57.64 | 57.29 | |
| CachEDBackbone=BART-base, Parameters=139M2025.01 | 48.9 | 14.4 | 19.8 | 64.1 | — | — | |
| SELECT & SUMMbase_model=Pegasus-X2024.04 | 48.29 | 11.4 | 46.62 | 57.31 | 57.39 | 57.2 | |
| Two-Stage Heuristicbase_model=LED2024.04 | 46.89 | 11.11 | 44.65 | 56.34 | 56.39 | 56.3 | |
| Full Textbase_model=Pegasus-X2024.04 | 46.2 | 10.58 | 45.35 | 57.05 | 57.1 | 57.01 | |
| Full Textbase_model=LED2024.04 | 46.15 | 10.62 | 44.46 | 56.04 | 56.32 | 55.77 | |
| SFTBackbone=LED-base, Parameters=162M2025.01 | 45.5 | 10.5 | 19.4 | 60.8 | — | — | |
| SLEDBackbone=BART-large, Parameters=406M2025.01 | 45.2 | 11.9 | 17.8 | 58.3 | — | — | |
| UnlimiformerBackbone=BART-base, Parameters=139M2025.01 | 44.8 | 12.3 | 18.3 | 58.7 | — | — | |
| SFTBackbone=BART-large, Parameters=406M2025.01 | 42.9 | 8.9 | 17.8 | 60.1 | — | — | |
| Random Selectionbase_model=LED, selection_ratio_k=25%2024.04 | 41.28 | 8.99 | 39.81 | 54.04 | 54.13 | 53.96 | |
| SFTBackbone=BART-base, Parameters=139M2025.01 | 39.2 | 8.2 | 17.2 | 57.8 | — | — | |
| CachEDBackbone=T5-large, Parameters=770M2025.01 | 36.7 | 7.7 | 17.6 | 56.9 | — | — | |
| SFTBackbone=T5-large, Parameters=770M2025.01 | 33.4 | 4.5 | 14.5 | 55.9 | — | — | |
| Unlimiformerattention_mechanism=retrieval-based2024.04 | 31.12 | 4.15 | 30.25 | 46.08 | 41.74 | 51.55 | |
| FLAN-UL2mode=Zero-Shot2024.04 | 25.86 | 5.42 | 24.71 | 50.71 | 51.4 | 48.79 | |
| SUMMNmode=Multi-Stage2024.04 | 22.65 | 3.01 | 22.42 | 39.68 | 40.87 | 38.6 | |
| GPT-3.5 Turbomode=Zero-Shot, model_variant=gpt-3.5-turbo-11062024.04 | 22.53 | 3.69 | 20.49 | 48.01 | 49.71 | 46.49 | |
| Lead-1024lead_length=10242024.04 | 17.69 | 2.1 | 16.78 | 48.1 | 49.43 | 46.86 | |
| Vicuna-13b-1.5mode=Zero-Shot2024.04 | 16.26 | 3.67 | 15.39 | 50.8 | 53.22 | 48.6 | |
| CreativeSumm2024.04 | 14.92 | 1.46 | 13.73 | 42.58 | 42.98 | 42.38 | |
| FLAN-T5-XXLmode=Zero-Shot2024.04 | 10.82 | 1.13 | 10.54 | 39.48 | 46.22 | 35.81 | |
| Lead-512lead_length=5122024.04 | 10.3 | 1.22 | 9.73 | 46.68 | 49.89 | 43.88 |