Text Summarization on Gigaword (test)
46.99ROUGE-1EncDec+DOC
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| EncDec+DOCi3=2, i2=22018.08 | 46.99 | 25.29 | 43.83 | — | — | |
| EncDec+DOCi3=22018.08 | 46.91 | 24.91 | 43.73 | — | — | |
| SEASS2018.08 | 46.86 | 24.58 | 43.53 | — | — | |
| EncDec2018.08 | 46.77 | 24.87 | 43.58 | — | — | |
| Kiyono et al.2018.08 | 46.34 | 24.85 | 43.49 | — | — | |
| BARTModel scale=base2021.01 | 39.34 | 20.07 | 41.25 | — | — | |
| PEGASUSSupervision level=Fully-supervised2022.05 | 39.12 | 19.86 | 36.24 | — | — | |
| T5Model scale=base2021.01 | 38.83 | 19.68 | 40.76 | — | — | |
| ProphetNetModel scale=large2021.01 | 38.49 | 18.41 | 39.84 | — | — | |
| Liu & PanEncoder=4-layer bidirectional LSTM, Decoder=1-layer LSTM, Attention=Softmax attention2017.04 | 38.22 | 18.7 | 35.74 | — | — | |
| BERT2BERTModel scale=base2021.01 | 38.16 | 18.89 | 40.06 | — | — | |
| Soft Monotonic AttentionEncoder=4-layer bidirectional LSTM, Decoder=1-layer LSTM2017.04 | 38.03 | 18.57 | 35.7 | — | — | |
| ABS2018.08 | 37.41 | 15.87 | 34.7 | — | — | |
| FTSumgContext Handling=Gating, priorities fact descriptions2017.11 | 37.27 | 17.65 | 34.24 | — | — | |
| Hard Monotonic AttentionEncoder=4-layer bidirectional LSTM, Decoder=1-layer LSTM2017.04 | 37.14 | 18 | 34.87 | — | — | |
| RNN+Attn2021.01 | 36.32 | 17.63 | 38.36 | — | — | |
| Suzuki & Nagata2017.04 | 36.3 | 17.31 | 33.88 | — | — | |
| Transformer2021.01 | 36.21 | 17.64 | 38.1 | — | — | |
| SEASSDecoding Strategy=beam2017.04 | 36.15 | 17.54 | 33.63 | — | — | |
| FTSumcContext Handling=Treats source sentence and relations equally2017.11 | 35.73 | 16.02 | 34.13 | — | — | |
| SEASSDecoding Strategy=greedy2017.04 | 35.48 | 16.5 | 32.93 | — | — | |
| words-lvt5k-1sentEvaluation protocol=Full length F1, Vocabulary size=5k, Summary length=1 sentence2016.02 | 35.3 | 16.64 | 32.62 | — | — | |
| Yu et al. 2016a2017.04 | 34.41 | 16.86 | 31.83 | — | — | |
| s2s+attArchitecture=Standard attentional s2s, Framework=dl4mt2017.11 | 34.23 | 15.52 | 31.57 | — | — | |
| Nallapati et al.2017.04 | 34.19 | 16.29 | 32.13 | — | — | |
| Miao & Blunsom2017.04 | 34.17 | 15.94 | 31.92 | — | — | |
| s2s+attDecoding Strategy=beam2017.04 | 34.04 | 15.95 | 31.68 | — | — | |
| RAS-ElmanEvaluation protocol=Full length F12016.02 | 33.78 | 15.97 | 31.15 | — | — | |
| Chopra et al.2017.04 | 33.78 | 15.97 | 31.15 | — | — | |
| CAs2sDecoding Strategy=beam2017.04 | 33.78 | 15.97 | 31.15 | — | — | |
| RAS-ElmanEncoder=Convolutional attention-based, Decoder=RNN2017.11 | 33.78 | 15.97 | 31.15 | — | — | |
| NACC (length control)Setting=Supervised, Model Category=NAR, Len (Avg chars)=34.4, Time (s)=0.0172022.05 | 33.66 | 13.73 | 31.79 | — | 4.74 | |
| s2s+attDecoding Strategy=greedy2017.04 | 33.18 | 14.79 | 30.8 | — | — | |
| NACC (truncate)Setting=Supervised, Model Category=NAR, Len (Avg chars)=34.15, Time (s)=0.0112022.05 | 33.12 | 13.93 | 31.34 | — | 1.34 | |
| CAs2sDecoding Strategy=greedy2017.04 | 33.1 | 14.45 | 30.25 | — | — | |
| Luong-NMTDecoding Strategy=beam2017.04 | 33.1 | 14.45 | 30.71 | — | — | |
| Luong-NMTArchitecture=Two-layer LSTMs, Hidden units=5002017.11 | 33.1 | 14.45 | 30.71 | — | — | |
| words-lvt2k-1sentEvaluation protocol=Full length F1, Vocabulary size=2k, Summary length=1 sentence2016.02 | 32.67 | 15.59 | 30.64 | 74.57 | — | |
| Feats2sDecoding Strategy=beam2017.04 | 32.67 | 15.59 | 30.64 | — | — | |
| Feats2sArchitecture=s2s RNN, Features=POS tag, NER2017.11 | 32.67 | 15.59 | 30.64 | — | — | |
| Su et al. (truncate)Setting=Supervised, Model Category=NAR, Len (Avg chars)=38.43, Time (s)=0.0162022.05 | 32.28 | 14.21 | 30.56 | — | 0 | |
| Qi et al. (truncate)Setting=Supervised, Model Category=NAR, Len (Avg chars)=27.98, Time (s)=0.0192022.05 | 31.69 | 12.52 | 30.05 | — | -2.79 | |
| ABS+type=Attention-Based Summarization, tuning=MERT tuning step (Z-MERT)2015.09 | 31 | 12.65 | 28.34 | 91.5 | — | |
| ABStype=Attention-Based Summarization2015.09 | 30.88 | 12.22 | 27.77 | 85.4 | — | |
| Yu et al. 2016b2017.04 | 30.27 | 13.68 | 27.91 | — | — | |
| ABS+Evaluation protocol=Full length F12016.02 | 29.78 | 11.89 | 26.97 | 91.5 | — | |
| ABS+Encoder=Attentive CNN, Decoder=NNLM, Features=Hand-crafted features2017.11 | 29.78 | 11.89 | 26.97 | — | — | |
| Rush et al.2017.04 | 29.76 | 11.88 | 26.96 | — | — | |
| ABS+Decoding Strategy=beam2017.04 | 29.76 | 11.88 | 26.96 | — | — | |
| ABSDecoding Strategy=beam2017.04 | 29.55 | 11.32 | 26.42 | — | — | |
| ABSEncoder=Attentive CNN, Decoder=NNLM2017.11 | 29.55 | 11.32 | 26.42 | — | — | |
| Yang et al. (truncate)Setting=Supervised, Model Category=NAR, Len (Avg chars)=35.372022.05 | 28.85 | 6.45 | 27 | — | -14.75 | |
| MOSES+2015.09 | 28.77 | 12.1 | 26.44 | 70.5 | — | |
| CITESTitleSupervision level=Zero-shot, Pre-training=Scientific domain + Paper titles2022.05 | 27.87 | 10.43 | 24.56 | — | — | |
| Zeng et al.2017.04 | 27.82 | 12.74 | 26.01 | — | — | |
| NACC (length control)Setting=Unsupervised, Model Category=NAR, Len (Avg chars)=47.03, Time (s)=0.0252022.05 | 27.45 | 8.87 | 25.14 | — | 5.19 | |
| NACC (truncate)Setting=Unsupervised, Model Category=NAR, Len (Avg chars)=47.77, Time (s)=0.0122022.05 | 25.79 | 8.94 | 23.75 | — | 2.21 | |
| TEDSupervision level=Unsupervised2022.05 | 25.58 | 8.94 | 22.83 | — | — | |
| SEQ³Supervision level=Unsupervised2022.05 | 25.39 | 8.21 | 22.68 | — | — | |
| Char-constrained searchSetting=Unsupervised, Model Category=Search, Len (Avg chars)=44.05, Time (s)=17.3242022.05 | 25.3 | 9.25 | 23.43 | — | 1.71 | |
| BART-LBSupervision level=Zero-shot2022.05 | 25.14 | 8.72 | 22.35 | — | — | |
| Schumann et al. (truncate)Setting=Unsupervised, Model Category=Search, Len (Avg chars)=45.45, Time (s)=9.5732022.05 | 24.98 | 9.08 | 23.18 | — | 0.97 | |
| CITESSupervision level=Zero-shot2022.05 | 24.75 | 8.42 | 21.84 | — | — | |
| Su et al. (truncate)Setting=Unsupervised, Model Category=NAR, Len (Avg chars)=45.24, Time (s)=0.0172022.05 | 24.65 | 8.64 | 22.98 | — | 0 | |
| Qi et al. (truncate)Setting=Unsupervised, Model Category=NAR, Len (Avg chars)=44.54, Time (s)=0.0192022.05 | 24.31 | 7.66 | 22.48 | — | -1.82 | |
| T5-LBSupervision level=Zero-shot2022.05 | 24 | 8.19 | 21.62 | — | — | |
| PEGASUSSupervision level=Zero-shot2022.05 | 23.39 | 7.59 | 20.2 | — | — | |
| PREFIX2015.09 | 23.14 | 8.25 | 21.73 | 100 | — | |
| BARTSupervision level=Zero-shot2022.05 | 22.07 | 7.47 | 20.02 | — | — | |
| Yang et al. (truncate)Setting=Unsupervised, Model Category=NAR, Len (Avg chars)=49.372022.05 | 21.7 | 4.6 | 20.13 | — | -9.84 | |
| BriefSupervision level=Unsupervised2022.05 | 21.26 | 5.6 | 18.89 | — | — | |
| Lead-50 charsSetting=Unsupervised, Model Category=Baseline, Len (Avg chars)=49.032022.05 | 20.66 | 7.08 | 19.3 | — | -9.23 | |
| COMPRESS2015.09 | 19.63 | 5.13 | 18.28 | 100 | — | |
| IR2015.09 | 16.91 | 5.55 | 15.58 | 29.2 | — | |
| T5Supervision level=Zero-shot2022.05 | 15.67 | 4.86 | 14.38 | — | — | |
| REFERENCE2015.09 | — | — | — | 45.6 | — |