Summarization on arXiv (test)
64.16ROUGE-1Top Down Transformer
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Top Down TransformerPooling Strategy=OracleAdaPool2022.03 | 64.16 | 33.39 | — | 56.88 | — | — | |
| ORACLE2021.07 | 60 | 30.6 | — | 53.03 | — | — | |
| ORACLESystem Category=Extractive2022.10 | 59.41 | 30.05 | — | 52.34 | — | — | |
| ORACLEModel Category=Extractive2024.08 | 59.41 | 30.05 | — | 52.34 | — | — | |
| ORACLEInput Sequence Length=28k tok.2022.03 | 53.97 | 26.42 | — | 48.12 | — | — | |
| OracleType=Upper Bound, Optimization=ROUGE-2 F12020.05 | 53.88 | 23.05 | — | 34.9 | — | — | |
| ORACLEInput Sequence Length=15k tok.2022.03 | 53.58 | 26.19 | — | 47.76 | — | — | |
| Top Down TransformerPooling Strategy=AdaPool, Number of Parameters=464M2022.03 | 50.95 | 21.93 | — | 45.61 | — | — | |
| EYEGLAXS (CHATGLM2-6B)Backbone=CHATGLM2-6B, Context Length=12K2024.08 | 49.02 | 21.01 | — | 43.33 | — | — | |
| EYEGLAXS (LLAMA2-7B)Backbone=LLAMA2-7B, Context Length=12K2024.08 | 48.96 | 21.07 | — | 43.3 | — | — | |
| LoBART(4k)+MCSType=Mix, Input sequence length=4k, ORCpad-rand=true2021.05 | 48.79 | 20.55 | — | 43.31 | — | — | |
| EYEGLAXS (LLAMA2-7B)Backbone=LLAMA2-7B, Context Length=4K2024.08 | 48.68 | 20.72 | — | 42.97 | — | — | |
| Top Down TransformerPooling Strategy=AvgPool, Number of Parameters=464M2022.03 | 48.67 | 20.7 | — | 43.91 | — | — | |
| GoSumModel Category=Extractive2024.08 | 48.61 | 20.53 | — | 42.8 | — | — | |
| Lodoss-full-LGSystem Category=Extractive, Loss Objective=Lsum + Lseg + bLDPP, Strategy=Local-Global context2022.10 | 48.45 | 20.72 | — | 42.55 | — | — | |
| Lodoss-full-LGModel Category=Extractive2024.08 | 48.45 | 20.72 | — | 42.55 | — | — | |
| MemSumMethod Type=Extractive2021.07 | 48.42 | 20.3 | — | 42.54 | — | — | |
| LongT5Model size=xl, Input length=16k, Attention mechanism=TGlobal2021.12 | 48.35 | 21.92 | — | 44.27 | — | — | |
| Long-T5Model Category=Abstractive2024.08 | 48.35 | 21.92 | — | 44.27 | — | — | |
| LongT5Model size=large, Input length=16k, Attention mechanism=TGlobal2021.12 | 48.28 | 21.63 | — | 44.11 | — | — | |
| Hepos-LSHMethod Type=Abstractive2021.07 | 48.24 | 20.26 | — | 41.78 | — | — | |
| LSHSource sequence length=71682021.04 | 48.24 | 20.26 | — | 41.78 | — | — | |
| Lodoss-fullSystem Category=Extractive, Loss Objective=Lsum + Lseg + bLDPP2022.10 | 48.2 | 20.5 | — | 42.28 | — | — | |
| Lodoss-jointSystem Category=Extractive, Loss Objective=Lsum + Lseg2022.10 | 47.97 | 20.13 | — | 42.03 | — | — | |
| Hepos-SinkhornMethod Type=Abstractive2021.07 | 47.87 | 20 | — | 41.5 | — | — | |
| BART(1k)+MCSType=Mix, Input sequence length=1k, TRC=true2021.05 | 47.68 | 19.77 | — | 42.25 | — | — | |
| Lodoss-baseSystem Category=Extractive, Loss Objective=Lsum2022.10 | 47.64 | 19.73 | — | 41.71 | — | — | |
| Lodoss-base (Longformer)Model Category=Extractive2024.08 | 47.64 | 19.73 | — | 41.71 | — | — | |
| PRIMERAEvaluation protocol=fully supervised2021.10 | 47.6 | 20.8 | — | 42.6 | — | — | |
| PRIMER2021.12 | 47.6 | 20.8 | — | 42.6 | — | — | |
| PRIMERAinput length limit=4k2021.10 | 47.58 | 20.75 | — | 42.57 | — | — | |
| NeuSumMethod Type=Extractive2021.07 | 47.49 | 21.56 | — | 41.58 | — | — | |
| CTRLsum(BART+BERT)Type=Mix2021.05 | 46.91 | 18.02 | — | 42.14 | — | — | |
| EYEGLAXS (CHATGLM2-6B)Backbone=CHATGLM2-6B, Context Length=4K2024.08 | 46.87 | 18.96 | — | 41.37 | — | — | |
| HATSystem Category=Abstractive2022.10 | 46.74 | 19.19 | — | 42.2 | — | — | |
| HAT-BARTBackbone=BART2021.12 | 46.68 | 19.07 | — | 42.17 | — | — | |
| HAT-BARTSource sequence length=30722021.04 | 46.68 | 19.07 | — | 42.17 | — | — | |
| BigBirdSequence length=4,0962020.04 | 46.63 | 19.02 | — | 41.77 | — | — | |
| LED-largeSequence length=16,3842020.04 | 46.63 | 19.62 | — | 41.83 | — | — | |
| BigBird(3k)Type=Abs, Input sequence length=3k2021.05 | 46.63 | 19.02 | — | 41.77 | — | — | |
| LED(16k)Type=Abs, Input sequence length=16k2021.05 | 46.63 | 19.62 | — | 41.83 | — | — | |
| BIGBIRD-PegasusBackbone=Pegasus2021.11 | 46.63 | 19.02 | — | 41.77 | — | — | |
| BigBirdNumber of Parameters=577M2022.03 | 46.63 | 19.02 | — | 41.77 | — | — | |
| LongformerNumber of Parameters=460M2022.03 | 46.63 | 19.62 | — | 41.83 | — | — | |
| BigBirdMethod Type=Abstractive2021.07 | 46.63 | 19.02 | — | 41.77 | — | — | |
| BigBird PEGASUSSummarization Strategy=Abstractive2022.03 | 46.63 | 19.02 | — | 41.77 | — | — | |
| LED-largeSummarization Strategy=Abstractive2022.03 | 46.63 | 19.62 | — | 41.48 | — | — | |
| Bigbird-PEGASUSinput length limit=3k2021.10 | 46.63 | 19.02 | — | 41.77 | — | — | |
| LEDinput length limit=16k2021.10 | 46.63 | 19.62 | — | 41.83 | — | — | |
| BigBird-PEGASUSModel size=large, Backbone=PEGASUS, Attention mechanism=BigBird2021.12 | 46.63 | 19.02 | — | 41.77 | — | — | |
| LEDModel size=large2021.12 | 46.63 | 19.62 | — | 41.83 | — | — | |
| BigBird-largeSystem Category=Abstractive, Model Scale=large2022.10 | 46.63 | 19.02 | — | 41.77 | — | — | |
| LED-16KSystem Category=Abstractive, Input Sequence Length=16K2022.10 | 46.63 | 19.62 | — | 41.83 | — | — | |
| Bigbird-largeModel Category=Abstractive2024.08 | 46.63 | 19.02 | — | 41.77 | — | — | |
| BigBirdSource sequence length=40962021.04 | 46.63 | 19.02 | — | 41.77 | — | — | |
| Beltagy et al. (2020)Evaluation protocol=fully supervised2021.10 | 46.6 | 19.6 | — | 41.8 | — | — | |
| LoBART(4k)Type=Abs, Input sequence length=4k, ORCpad-rand=true2021.05 | 46.59 | 18.72 | — | 41.24 | — | — | |
| Transformer-BARTSource sequence length=30722021.04 | 46.54 | 18.82 | — | 42 | — | — | |
| TLM-I+E (G,G)Type=Oracle, train_extracts=Ground-truth, test_extracts=Ground-truth2019.09 | 46.4 | 18.15 | 8.71 | 42.27 | — | — | |
| TerraformerBeam search=No, Greedy decoding=Yes, Pre-training=C42021.11 | 45.4 | 17.86 | — | 41.21 | 26.33 | — | |
| HiStruct+Backbone=Longformer-base, Input Sequence Length=28k tok., Embedding Injection Configuration=sHE+STE2022.03 | 45.22 | 17.67 | — | 40.16 | — | — | |
| Histruct+Model Category=Extractive2024.08 | 45.22 | 17.67 | — | 40.16 | — | — | |
| HiStruct+Backbone=Longformer-base, Input Sequence Length=28k tok., Embedding Injection Configuration=sHE+STE(classified)2022.03 | 45.17 | 17.61 | — | 40.1 | — | — | |
| HiStruct+Backbone=Longformer-base, Input Sequence Length=15k tok., Embedding Injection Configuration=sHE+STE2022.03 | 45.02 | 17.48 | — | 39.94 | — | — | |
| DANCERType=Abs2021.05 | 45.01 | 17.6 | — | 40.56 | — | — | |
| DANCER PEGASUSBackbone=Pegasus2021.11 | 45.01 | 17.6 | — | 40.56 | — | — | |
| Dancer2022.03 | 45.01 | 17.6 | — | 40.56 | — | — | |
| DancerMethod Type=Abstractive2021.07 | 45.01 | 17.6 | — | 40.56 | — | — | |
| DANCER PEGASUSSummarization Strategy=Abstractive2022.03 | 45.01 | 17.6 | — | 40.56 | — | — | |
| DANCER PEGASUSBackbone=PEGASUS2021.12 | 45.01 | 17.6 | — | 40.56 | — | — | |
| BART(1k)Type=Abs, Input sequence length=1k2021.05 | 44.96 | 17.25 | — | 39.76 | — | — | |
| HiStruct+Backbone=Longformer-base, Input Sequence Length=15k tok., Embedding Injection Configuration=sHE+STE(classified)2022.03 | 44.94 | 17.42 | — | 39.9 | — | — | |
| SSN-DM2022.03 | 44.9 | 19.06 | — | 32.77 | — | — | |
| PEGASUSSummarization Strategy=Abstractive2022.03 | 44.7 | 17.27 | — | 25.8 | — | — | |
| LED-largeSequence length=4,0962020.04 | 44.4 | 17.94 | — | 39.76 | — | — | |
| LED(4k)Type=Abs, Input sequence length=4k2021.05 | 44.4 | 17.94 | — | 39.76 | — | — | |
| LEDinput length limit=4k2021.10 | 44.4 | 17.94 | — | 39.76 | — | — | |
| LED-4KSystem Category=Abstractive, Input Sequence Length=4K2022.10 | 44.4 | 17.94 | — | 39.76 | — | — | |
| Gold ExtType=Oracle2019.09 | 44.25 | 18.17 | 9.14 | 35.33 | — | — | |
| Pegasus2020.04 | 44.21 | 16.95 | — | 38.83 | — | — | |
| PegasusType=Abs2021.05 | 44.21 | 16.95 | — | 38.83 | — | — | |
| Pegasus LargePre-training=C42021.11 | 44.21 | 16.95 | — | 38.83 | 25.67 | — | |
| PegasusNumber of Parameters=568M2022.03 | 44.21 | 16.95 | — | 38.83 | — | — | |
| PEGASUSMethod Type=Abstractive2021.07 | 44.21 | 16.95 | — | 38.83 | — | — | |
| PEGASUSinput length limit=1k2021.10 | 44.21 | 16.95 | — | 38.83 | — | — | |
| PEGASUSSource sequence length=40962021.04 | 44.21 | 16.95 | — | 38.83 | — | — | |
| ExtSum-LG+RdType=Ext2021.05 | 44.01 | 17.79 | — | 39.09 | — | — | |
| ExtSum-LG + RLossSummarization Strategy=Extractive2022.03 | 44.01 | 17.79 | — | 39.09 | — | — | |
| ExtSum-LG + RdLossSystem Category=Extractive, Loss Objective=Redundancy Loss2022.10 | 44.01 | 17.79 | — | 39.09 | — | — | |
| BaselineBackbone=BART-base2023.05 | 43.95 | — | — | — | — | — | |
| ExtSum-LG + MMR-Select+Summarization Strategy=Extractive2022.03 | 43.87 | 17.5 | — | 38.97 | — | — | |
| GlobalLocalContLearning Paradigm=Supervised, Method Type=Extractive2020.05 | 43.62 | 17.36 | — | 29.14 | — | — | |
| Atten-ContMethod Type=Extractive2021.07 | 43.62 | 17.36 | — | 29.14 | — | — | |
| ExtSum-LGSystem Category=Extractive2022.10 | 43.62 | 17.36 | — | 29.14 | — | — | |
| HiStruct+Backbone=Longformer-base, Input Sequence Length=15k tok., Embedding Injection Configuration=sHE2022.03 | 43.04 | 15.87 | — | 38.13 | — | — | |
| SRformerBackbone=BART-base2023.05 | 42.99 | 14.97 | — | 25.29 | — | 37.86 | |
| SummaRuNNerLearning Paradigm=Supervised, Method Type=Extractive2020.05 | 42.81 | 16.52 | — | 28.23 | — | — | |
| SummaRuNNerMethod Type=Extractive2021.07 | 42.81 | 16.52 | — | 28.23 | — | — | |
| Dancer2020.04 | 42.7 | 16.54 | — | 38.44 | — | — | |
| DANCER RUM2021.11 | 42.7 | 16.54 | — | 38.44 | — | — |