Summarization on Arxiv
23.05ROUGE-2ORACLE
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ORACLE2022.08 | 23.05 | 53.88 | 44.9 | — | |
| LongT5Model Size=xl, Input Length=16k, Attention=TGlobal2021.12 | 21.92 | 48.35 | 44.27 | — | |
| LongT5Model Size=large, Input Length=16k, Attention=TGlobal2021.12 | 21.63 | 48.28 | 44.11 | — | |
| LongT5Model Size=xl, Input Length=8k, Attention=TGlobal2021.12 | 20.84 | 47.44 | 43.34 | — | |
| PRIMER2021.12 | 20.8 | 47.6 | 42.6 | — | |
| GRETEL2022.08 | 20.31 | 48.17 | 42.84 | — | |
| MemSum2026.02 | 20.17 | 48.23 | 42.31 | — | |
| GloSA-sum2026.02 | 20 | 47.5 | 42 | — | |
| PEGASUS2026.02 | 19.7 | 43.27 | 34.79 | — | |
| LongT5Model Size=large, Input Length=8k, Attention=TGlobal2021.12 | 19.67 | 46.61 | 42.44 | — | |
| BERTSum+SW2022.08 | 19.67 | 47.86 | 42.49 | — | |
| LEDModel Size=large2021.12 | 19.62 | 46.63 | 41.83 | — | |
| LongT5Model Size=xl, Input Length=4k, Attention=TGlobal2021.12 | 19.51 | 45.99 | 42.04 | — | |
| GBT-EXTSUM2022.08 | 19.21 | 48.08 | 42.68 | — | |
| HAT-BART2021.12 | 19.07 | 46.68 | 42.17 | — | |
| BIGBIRD-PegasusModel Category=Large2020.07 | 19.02 | 46.63 | 41.77 | — | |
| BigBird-PEGASUSModel Size=large2021.12 | 19.02 | 46.63 | 41.77 | — | |
| BigBird2026.02 | 19.02 | 46.63 | 41.77 | — | |
| LongT5Model Size=large, Input Length=4k, Attention=TGlobal2021.12 | 18.6 | 45.64 | 41.51 | — | |
| LongT5Model Size=base, Input Length=4k, Attention=TGlobal2021.12 | 18.54 | 44.87 | 40.97 | — | |
| T5.1.1 + PSGModel Size=large, Input Length=4k2021.12 | 18.4 | 45.86 | 41.62 | — | |
| T5.1.1Model Size=large, Input Length=4k2021.12 | 18.2 | 44.51 | 40.62 | — | |
| BERTSum2026.02 | 18.2 | 47.1 | 20.8 | — | |
| DANCER PEGASUS2021.12 | 17.6 | 45.01 | 40.56 | — | |
| DANCER2026.02 | 17.6 | 45.01 | 40.56 | — | |
| PegasusModel Category=Large, Evaluation Source=Reported2020.07 | 16.95 | 44.21 | 38.83 | — | |
| Longformer-Ext2022.08 | 16.88 | 45.24 | 40.03 | — | |
| PegasusModel Category=Large, Evaluation Source=Re-eval2020.07 | 16.83 | 43.85 | 39.17 | — | |
| T5.1.1Model Size=large, Input Length=2k2021.12 | 16.62 | 42.84 | 39.01 | — | |
| BART2026.02 | 16.55 | 43.84 | 39.86 | — | |
| DancerModel Category=Prior Art2020.07 | 16.54 | 42.7 | 38.44 | — | |
| T5.1.1 + PSGModel Size=large, Input Length=2k2021.12 | 16.5 | 42.85 | 38.99 | — | |
| BIGBIRD-ROBERTaModel Category=Base2020.07 | 16.43 | 41.22 | 36.96 | — | |
| UNISUMMFew-shot samples=1002022.11 | 16.42 | 44.33 | 39.71 | — | |
| MultiBARTFew-shot samples=1002022.11 | 15.97 | 43.56 | 39.01 | — | |
| OursFew-shot examples=1002021.10 | 15.85 | 43.42 | 24.07 | — | |
| Sent-PTRModel Category=Prior Art2020.07 | 15.63 | 42.32 | 38.06 | — | |
| UNISUMMFew-shot samples=102022.11 | 15.38 | 43.33 | 38.69 | — | |
| Extr-Abst-TLMModel Category=Prior Art2020.07 | 14.69 | 41.62 | 38.03 | — | |
| Reformer-Ext2022.08 | 14.68 | 43.26 | 38.1 | — | |
| MultiBARTFew-shot samples=102022.11 | 14.68 | 41.45 | 37.01 | — | |
| T5.1.1Model Size=large, Input Length=1k2021.12 | 14.02 | 39.79 | 36.23 | — | |
| OursFew-shot examples=102021.10 | 13.81 | 41.13 | 23.02 | — | |
| LEDFew-shot examples=1002021.10 | 13.74 | 41 | 22.34 | — | |
| T5.1.1 + PSGModel Size=large, Input Length=1k2021.12 | 13.61 | 38.53 | 35.08 | — | |
| BERTSum2022.08 | 13.01 | 41.24 | 36.1 | — | |
| BART-PTFew-shot samples=1002022.11 | 11.46 | 38.03 | 34.2 | — | |
| LEDFew-shot examples=102021.10 | 11.16 | 36.51 | 20.68 | — | |
| Long-Doc-Seq2SeqModel Category=Prior Art2020.07 | 11.05 | 35.8 | 31.8 | — | |
| GPT-3.5 (text-davinci-002)Few-shot samples=12022.11 | 10.83 | — | — | — | |
| BARTFew-shot examples=1002021.10 | 10.78 | 37.62 | 20.99 | — | |
| LexRankModel Category=Prior Art2020.07 | 10.73 | 33.85 | 28.99 | — | |
| Transformer + PegasusModel Category=Base2020.07 | 10.16 | 34.81 | 30.14 | — | |
| PEGASUSFew-shot samples=1002022.11 | 10.14 | 38.08 | 31.06 | — | |
| BART-FTFew-shot samples=1002022.11 | 10.07 | 36.69 | 32.67 | — | |
| PEGASUSFew-shot examples=1002021.10 | 9.55 | 36.38 | 20.83 | — | |
| Pntr-Gen-Seq2SeqModel Category=Prior Art2020.07 | 9.04 | 32.06 | 25.16 | — | |
| LEAD2022.08 | 8.94 | 33.66 | 22.19 | — | |
| TextRank2026.02 | 8.8 | 33.1 | 30.05 | — | |
| Sent-CLFModel Category=Prior Art2020.07 | 8.71 | 34.01 | 30.41 | — | |
| BARTFew-shot examples=102021.10 | 8.7 | 32.53 | 17.98 | — | |
| PEGASUSFew-shot examples=102021.10 | 8.52 | 33.1 | 19.4 | — | |
| Topic-GraphSum2022.08 | 8.52 | 32.41 | 18.52 | — | |
| PEGASUSFew-shot samples=102022.11 | 8.46 | 34.81 | 29.12 | — | |
| Transformer + ROBERTAModel Category=Base2020.07 | 8.13 | 31.98 | 29.53 | — | |
| BART-PTFew-shot samples=102022.11 | 8.08 | 29.85 | 26.76 | — | |
| PEGASUSFew-shot examples=02021.10 | 7.94 | 29.76 | 17.27 | — | |
| BARTFew-shot examples=02021.10 | 7.57 | 23.26 | 12.01 | — | |
| LSAModel Category=Prior Art2020.07 | 7.42 | 29.91 | 25.67 | — | |
| SumBasicModel Category=Prior Art2020.07 | 6.95 | 29.47 | 26.3 | — | |
| TransformerModel Category=Base2020.07 | 6.7 | 28.52 | 25.58 | — | |
| Attn-Seq2SeqModel Category=Prior Art2020.07 | 6 | 29.3 | 25.56 | — | |
| Lead-32026.02 | 5.98 | 25.53 | 15.22 | — | |
| BART-FTFew-shot samples=102022.11 | 4.98 | 28.4 | 25.15 | — | |
| OursFew-shot examples=02021.10 | 4.6 | 15.8 | 9.13 | — | |
| LEDFew-shot examples=02021.10 | 3.76 | 13.94 | 8.35 | — | |
| BART-PTNumber of shots=102022.11 | — | — | — | 1.08 | |
| BART-PTNumber of shots=1002022.11 | — | — | — | 0.83 | |
| MultiBARTNumber of shots=102022.11 | — | — | — | 0.32 | |
| MultiBARTNumber of shots=1002022.11 | — | — | — | 0.64 | |
| PEGASUSNumber of shots=102022.11 | — | — | — | 0.52 | |
| PEGASUSNumber of shots=1002022.11 | — | — | — | 0.55 | |
| UNISUMMNumber of shots=102022.11 | — | — | — | 0.93 | |
| UNISUMMNumber of shots=1002022.11 | — | — | — | 0.54 |