Abstractive Text Summarization on CNN/Daily Mail (test)
50.77ROUGE-LMax
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| MaxStrategy=Oracle selection (maximum ROUGE), Base generation model=BART (facebook/bart-large-cnn)2021.06 | 50.77 | 54.36 | 28.73 | — | — | 70.77 | 61.67 | — | — | — | |
| JGR-R (init w. BRIO)Type=Ranker (re-rank outputs of JGR-G), Initialization=BRIO2022.06 | 46.56 | 48.86 | 23.35 | — | 39.59 | — | — | — | — | — | |
| JGR-G (init w. BRIO)Type=Generator, Initialization=BRIO2022.06 | 46.11 | 48.39 | 23.22 | — | 39.24 | — | — | — | — | — | |
| BRIOUsage=Backbone model for JGR-G2022.06 | 44.57 | 47.48 | 23.55 | — | 38.53 | — | — | — | — | — | |
| JGR-RType=Ranker (re-rank outputs of JGR-G), Generator Backbone=BART-large, Ranker Backbone=RoBERTa-large2022.06 | 44.5 | 47.63 | 23.59 | — | 38.57 | — | — | — | — | — | |
| JGR-GType=Generator, Backbone=BART-large2022.06 | 43.74 | 46.86 | 23.18 | — | 37.93 | — | — | — | — | — | |
| SimCLSBase generation model=BART (facebook/bart-large-cnn)2021.06 | 43.54 | 46.67 | 22.15 | — | — | 66.14 | 59.31 | — | — | — | |
| GSumSource=Original paper results2021.06 | 42.48 | 45.94 | 22.32 | — | — | — | — | — | — | — | |
| GSUM2022.06 | 42.48 | 45.94 | 22.32 | — | 36.91 | — | — | — | — | — | |
| ERNIE-GENLARGEData=430G, Params=340M2020.01 | 41.6 | 44.31 | 21.35 | — | — | — | — | — | — | — | |
| PALMModel size=LARGE2020.04 | 41.41 | 44.3 | 21.12 | — | — | — | — | — | — | — | |
| ProphetNet2021.05 | 41.3 | 44.2 | 21.17 | — | — | — | — | — | — | — | |
| ProphetNetSource=Original paper results2021.06 | 41.3 | 44.2 | 21.17 | — | — | — | — | — | — | — | |
| ProphetNet2022.06 | 41.3 | 44.2 | 21.17 | — | — | — | — | — | — | — | |
| ProphetNet2022.06 | 41.3 | 44.2 | 21.17 | — | 35.56 | — | — | — | — | — | |
| OriginBase generation model=BART (facebook/bart-large-cnn)2021.06 | 41.28 | 44.39 | 21.21 | — | — | 64.67 | 58.67 | — | — | — | |
| ERNIE-GENLARGEData=16G, Params=340M2020.01 | 41.26 | 44.02 | 21.17 | — | — | — | — | — | — | — | |
| ERNIE-GENModel size=LARGE2020.04 | 41.26 | 44.02 | 21.17 | — | — | — | — | — | — | — | |
| ERNIE-GEN-largeModel Size=large2022.06 | 41.26 | 44.02 | 21.17 | — | — | — | — | — | — | — | |
| BART-largeTraining Objective=DITTO2022.06 | 41.16 | 44.41 | 21.45 | — | — | — | — | — | — | — | |
| PALM2022.06 | 41.14 | 44.3 | 21.12 | — | — | — | — | — | — | — | |
| PEGASUS (HugeNews)Data=3.8T, Params=568M2020.01 | 41.11 | 44.17 | 21.47 | — | — | — | — | — | — | — | |
| PEGASUSLARGEModel Size=Large, Pre-training Corpus=HugeNews2019.12 | 41.11 | 44.17 | 21.47 | — | — | — | — | — | — | — | |
| PEGASUS2020.04 | 41.11 | 44.17 | 21.47 | — | — | — | — | — | — | — | |
| PEGASUSPre-training Corpus=HugeNews2021.05 | 41.11 | 44.17 | 21.47 | — | — | — | — | — | — | — | |
| PegasusSource=Original paper results2021.06 | 41.11 | 44.17 | 21.47 | — | — | — | — | — | — | — | |
| PEGASUS2022.06 | 41.11 | 44.17 | 21.47 | — | — | — | — | — | — | — | |
| PEGASUS2022.06 | 41.11 | 44.17 | 21.47 | — | 35.58 | — | — | — | — | — | |
| RandomStrategy=Oracle selection (random candidate), Base generation model=BART (facebook/bart-large-cnn)2021.06 | 40.94 | 43.98 | 20.06 | — | — | 64.65 | 58.6 | — | — | — | |
| BARTLARGEData=160G, Params=400M2020.01 | 40.9 | 44.16 | 21.28 | — | — | — | — | — | — | — | |
| Previous SOTA2019.12 | 40.9 | 44.16 | 21.28 | — | — | — | — | — | — | — | |
| BARTModel size=LARGE2020.04 | 40.9 | 44.16 | 21.28 | — | — | — | — | — | — | — | |
| BART2021.05 | 40.9 | 44.16 | 21.28 | — | — | — | — | — | — | — | |
| BARTSource=Original paper results2021.06 | 40.9 | 44.16 | 21.28 | — | — | — | — | — | — | — | |
| BARTSize=Large2021.03 | 40.9 | 44.2 | 21.3 | — | — | — | — | — | — | — | |
| BARTBackbone=BART-large, Implementation Source=Our Implementation2022.06 | 40.9 | 44.16 | 21.28 | — | 35.45 | — | — | — | — | — | |
| BART-largeTraining Objective=SG2022.06 | 40.89 | 44.18 | 21.17 | — | — | — | — | — | — | — | |
| BART-largeTraining Objective=MLE2022.06 | 40.83 | 44.11 | 21.21 | — | — | — | — | — | — | — | |
| BART-largeTraining Objective=UL-token2022.06 | 40.83 | 44.17 | 21.2 | — | — | — | — | — | — | — | |
| PEGASUS(C4)Data=750G, Params=568M2020.01 | 40.76 | 43.9 | 21.2 | — | — | — | — | — | — | — | |
| PEGASUSLARGEModel Size=Large, Pre-training Corpus=C42019.12 | 40.76 | 43.9 | 21.2 | — | — | — | — | — | — | — | |
| PEGASUSPre-training Corpus=C42021.05 | 40.76 | 43.9 | 21.2 | — | — | — | — | — | — | — | |
| ProphetNet2020.01 | 40.72 | 43.68 | 20.64 | — | — | — | — | — | — | — | |
| BART-largeTraining Objective=UL-token+seq2022.06 | 40.71 | 44.13 | 21.15 | — | — | — | — | — | — | — | |
| T5XLARGEData=750G, Params=11B2020.01 | 40.69 | 43.52 | 21.55 | — | — | — | — | — | — | — | |
| T52021.05 | 40.69 | 43.52 | 21.55 | — | — | — | — | — | — | — | |
| T5-11BParameters=11 billion2019.10 | 40.69 | 43.52 | 21.55 | — | — | — | — | — | — | — | |
| Previous best2019.10 | 40.63 | 43.47 | 20.3 | — | — | — | — | — | — | — | |
| UniLMModel size=LARGE2020.04 | 40.51 | 43.33 | 20.21 | — | — | — | — | — | — | — | |
| UniLM2020.01 | 40.51 | 43.33 | 20.21 | — | — | — | — | — | — | — | |
| UniLM2021.05 | 40.51 | 43.33 | 20.21 | — | — | — | — | — | — | — | |
| GLMBackbone=RoBERTa2021.03 | 40.5 | 43.8 | 21 | — | — | — | — | — | — | — | |
| UniLM2022.06 | 40.34 | 43.08 | 20.43 | — | — | — | — | — | — | — | |
| UniLM V22022.06 | 40.14 | 43.16 | 20.42 | — | — | — | — | — | — | — | |
| UniLMv2Size=Base2021.03 | 40.1 | 43.2 | 20.4 | — | — | — | — | — | — | — | |
| T5-3BParameters=3 billion2019.10 | 39.94 | 42.72 | 21.02 | — | — | — | — | — | — | — | |
| PEGFAME2021.05 | 39.9 | 42.95 | 20.79 | — | — | — | — | — | — | — | |
| T5Size=Large2021.03 | 39.8 | 42.5 | 20.7 | — | — | — | — | — | — | — | |
| T5LARGEData=750G, Params=340M2020.01 | 39.75 | 42.5 | 20.68 | — | — | — | — | — | — | — | |
| T5Model size=LARGE2020.04 | 39.75 | 42.5 | 20.68 | — | — | — | — | — | — | — | |
| T5-LargeParameters=770 million2019.10 | 39.75 | 42.5 | 20.68 | — | — | — | — | — | — | — | |
| PALM2020.04 | 39.71 | 42.71 | 19.97 | — | — | — | — | — | — | — | |
| PEGASUSimplementation=ours2021.05 | 39.61 | 42.62 | 20.38 | — | — | — | — | — | — | — | |
| SAGCopyArchitecture=Transformer-based2021.05 | 39.44 | 42.53 | 19.92 | — | — | — | — | — | — | — | |
| T5-BaseParameters=220 million2019.10 | 39.4 | 42.05 | 20.34 | — | — | — | — | — | — | — | |
| BERTSUMEXTABSmode=extractive-abstractive2020.01 | 39.18 | 42.13 | 19.6 | — | — | — | — | — | — | — | |
| BertSum2022.06 | 39.18 | 42.13 | 19.6 | — | — | — | — | — | — | — | |
| MASS2020.04 | 39.01 | 42.12 | 19.5 | — | — | — | — | — | — | — | |
| MASS2020.01 | 39.01 | 42.12 | 19.5 | — | — | — | — | — | — | — | |
| MASS2021.05 | 39.01 | 42.12 | 19.5 | — | — | — | — | — | — | — | |
| PEGASUSBASEModel Size=Base2019.12 | 38.93 | 41.79 | 18.81 | — | — | — | — | — | — | — | |
| BERTSumAbsType=Abstractive2021.03 | 38.8 | 41.7 | 19.4 | — | — | — | — | — | — | — | |
| SELECTORdecoder=10-Beam PG, K (number of mixtures)=12019.09 | 38.79 | 41.72 | 18.74 | — | — | — | — | — | — | — | |
| BERTSUMABS2020.04 | 38.76 | 41.72 | 19.39 | — | — | — | — | — | — | — | |
| BERTSUMABSmode=abstractive2020.01 | 38.76 | 41.72 | 19.39 | — | — | — | — | — | — | — | |
| RK4-block2022.03 | 38.68 | 41.83 | 18.84 | — | — | — | — | — | — | — | |
| S2S-ELMOembeddings=ELMO2020.01 | 38.47 | 41.56 | 18.94 | — | — | — | — | — | — | — | |
| RK2-block2022.03 | 38.41 | 41.58 | 18.57 | — | — | — | — | — | — | — | |
| T5-SmallParameters=60 million2019.10 | 38.35 | 41.12 | 19.56 | — | — | — | — | — | — | — | |
| Bottom-Up2019.09 | 38.34 | 41.22 | 18.68 | — | — | — | — | — | — | — | |
| Bottom-Up2020.01 | 38.34 | 41.22 | 18.68 | — | — | — | — | — | — | — | |
| Bottom-Up2021.05 | 38.34 | 41.22 | 18.68 | — | — | — | — | — | — | — | |
| Bottom-Up Summarization2022.03 | 38.34 | 41.22 | 18.68 | — | — | — | — | — | — | — | |
| BOTTOM-UP2022.12 | 38.34 | 41.22 | 18.68 | — | — | — | — | — | — | — | |
| EIT2022.12 | 38.33 | 41.62 | 18.7 | — | — | — | — | — | — | — | |
| Transformer#Params=44.1M, #MAdds (30)=2.0G, #MAdds (100)=3.6G, #MAdds (1000)=29.9G2020.04 | 38.3 | 41.4 | 18.9 | — | — | — | — | — | — | — | |
| Lite Transformer#Params=17.3M, #MAdds (30)=0.8G, #MAdds (100)=1.5G, #MAdds (1000)=12.5G2020.04 | 38.3 | 41.3 | 18.8 | — | — | — | — | — | — | — | |
| E-EIT2022.12 | 38.28 | 41.58 | 18.63 | — | — | — | — | — | — | — | |
| Talking-Head2022.12 | 38.06 | 41.26 | 18.34 | — | — | — | — | — | — | — | |
| NEUSUM2022.03 | 37.98 | 41.59 | 19.01 | — | — | — | — | — | — | — | |
| DCA2019.09 | 37.92 | 41.69 | 19.47 | — | — | — | — | — | — | — | |
| Soft Fusion2022.03 | 37.9 | 41 | 18.3 | — | — | — | — | — | — | — | |
| SURFACE2022.12 | 37.9 | 41 | 18.3 | — | — | — | — | — | — | — | |
| DMAN2021.03 | 37.88 | 40.98 | 18.29 | — | 32.38 | — | — | — | — | — | |
| Mask Attention Network2022.06 | 37.88 | 40.98 | 18.29 | — | — | — | — | — | — | — | |
| DMAN2022.12 | 37.88 | 40.98 | 18.29 | — | — | — | — | — | — | — | |
| Transformervariant=baseline2022.12 | 37.58 | 40.84 | 18 | — | — | — | — | — | — | — | |
| ROBFAME2021.05 | 37.51 | 40.27 | 18.43 | — | — | — | — | — | — | — | |
| Transformer + LayerDropEnc=6, Dec=82019.09 | 37.5 | 41.1 | 18.1 | — | — | — | — | — | — | — | |
| Residual-block2022.03 | 37.29 | 40.47 | 17.73 | — | — | — | — | — | — | — |