Abstractive Summarization on CNN/Daily Mail non-anonymous (test)
44.16ROUGE-1BART_LARGE
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| BART_LARGE#Param=400M, Model size=LARGE2020.02 | 44.16 | 21.28 | 40.9 | — | |
| T5_11B#Param=11B, Model size=LARGE2020.02 | 43.52 | 21.55 | 40.69 | — | |
| UNILMv2_BASE#Param=110M, Model size=BASE, relative position bias=false2020.02 | 43.45 | 20.71 | 40.49 | — | |
| UNILMv2_BASE#Param=110M, Model size=BASE2020.02 | 43.16 | 20.42 | 40.14 | — | |
| UNILM_LARGE#Param=340M, Model size=LARGE2020.02 | 43.08 | 20.43 | 40.34 | — | |
| MASS_BASE#Param=123M, Model size=BASE2020.02 | 42.12 | 19.5 | 39.01 | — | |
| T5_BASE#Param=220M, Model size=BASE2020.02 | 42.05 | 20.34 | 39.4 | — | |
| BERTSUMABS#Param=156M, Model size=BASE2020.02 | 41.72 | 19.39 | 38.76 | — | |
| Celikyilmaz et al. (2018)Architecture=LSTM-based, Optimization=Reinforcement learning2019.05 | 41.69 | 19.47 | 37.92 | — | |
| Bottom-Up Summarizationbeam size=10, backbone=Pointer-Generator2018.08 | 41.22 | 18.68 | 38.34 | — | |
| Bottom-Up Summarization (CopyTransformer)beam size=10, backbone=CopyTransformer2018.08 | 40.96 | 18.38 | 38.16 | — | |
| Sentence Rewriting2018.08 | 40.88 | 17.8 | 38.54 | — | |
| Inconsistency loss2018.08 | 40.68 | 17.97 | 37.13 | — | |
| ROUGESal+Ent (RL)Training Objective=Reinforce mixed loss (XE+RL), Reward Function=ROUGE-L + Saliency + Entailment2018.04 | 40.43 | 18 | 37.1 | 20.02 | |
| Saliency + Entailment reward2018.08 | 40.43 | 18 | 37.1 | — | |
| LEAD-3Pre-training=None2020.02 | 40.42 | 17.62 | 36.67 | — | |
| ROUGE+Ent (RL)Training Objective=Reinforce mixed loss (XE+RL), Reward Function=ROUGE-L + Entailment2018.04 | 40.37 | 17.89 | 37.13 | 19.94 | |
| ROUGESal (RL)Training Objective=Reinforce mixed loss (XE+RL), Reward Function=ROUGE-L + Saliency2018.04 | 40.36 | 17.97 | 37 | 19.84 | |
| ROUGE (RL)Training Objective=Reinforce mixed loss (XE+RL), Reward Function=ROUGE-L2018.04 | 39.99 | 17.72 | 36.66 | 18.93 | |
| Paulus (2017) (RL)Dataset Version=anonymous, Training Objective=Reinforce mixed loss (XE+RL)2018.04 | 39.87 | 15.82 | 36.9 | — | |
| ML + RL2018.08 | 39.87 | 15.82 | 36.9 | — | |
| Transformer LMModel layers=12, Pre-training=Full2019.05 | 39.65 | 17.74 | 36.85 | — | |
| See et al. (2017)Dataset Version=non-anonymous2018.04 | 39.53 | 17.28 | 36.38 | 18.72 | |
| Entail (RL)Training Objective=Reinforce mixed loss (XE+RL), Reward Function=Entailment2018.04 | 39.53 | 17.51 | 36.44 | 20.15 | |
| Pointer-Generator + Coveragemechanism=full coverage mechanism2018.08 | 39.53 | 17.28 | 36.38 | — | |
| PTRNETPre-training=None2020.02 | 39.53 | 17.28 | 36.38 | — | |
| Baseline (XE)Training Objective=Cross-entropy loss (XE)2018.04 | 39.41 | 17.33 | 36.07 | 18.27 | |
| Fore-copy-unkDataset Version=Non-anonymized, Encoder Architecture=Transformer2021.12 | 39.31 | 17.13 | 36.25 | — | |
| CopyTransformer + Coverage Penaltycoverage penalty beta=10, backbone=Transformer with copy-attention2018.08 | 39.25 | 17.54 | 36.45 | — | |
| CopyTransformerModel layers=4, Copy mechanism=true, Coverage loss=true2019.05 | 39.25 | 17.54 | 36.45 | — | |
| Encoder-DecoderModel layers=12, Pre-training=Full2019.05 | 39.18 | 17 | 36.33 | — | |
| Pointer-Generator + Coverage Penaltycoverage penalty beta=102018.08 | 39.12 | 17.35 | 36.12 | — | |
| Key information guide network2018.08 | 38.95 | 17.12 | 35.68 | — | |
| Force-copyDataset Version=Non-anonymized, Encoder Architecture=Transformer2021.12 | 38.76 | 16.84 | 35.42 | — | |
| Pointer-GeneratorDataset Version=Non-anonymized, Encoder Architecture=Transformer, Implementation Source=Re-implemented, Reference Source=See et al. (2017)2021.12 | 38.66 | 16.97 | 35.61 | — | |
| Pointer-Generator + DiffMask2018.08 | 38.45 | 16.88 | 35.81 | — | |
| Paulus (2017) (XE)Dataset Version=anonymous, Training Objective=Cross-entropy loss (XE)2018.04 | 38.3 | 14.81 | 35.49 | — | |
| ML + Intra-Attention2018.08 | 38.3 | 14.81 | 35.49 | — | |
| ML + Intra-AttentionDataset Version=Non-anonymized, Reference Source=Paulus et al. (2018)2021.12 | 38.3 | 14.81 | 35.49 | — | |
| BidirEncoder-DecoderModel layers=4, Pre-training=None2019.05 | 37.74 | 16.27 | 34.76 | — | |
| Transformer LMModel layers=12, Pre-training=None2019.05 | 37.72 | 16.14 | 34.62 | — | |
| Pointer-Generator + Mask Onlymechanism=increased supervision on copy2018.08 | 37.7 | 15.63 | 35.49 | — | |
| Pointer-Generator + Multi-Task2018.08 | 37.67 | 15.59 | 35.47 | — | |
| Encoder-DecoderModel layers=12, Pre-training=None2019.05 | 36.72 | 15.22 | 33.84 | — | |
| Pointer-Generatorcoverage=false2018.08 | 36.44 | 15.66 | 33.42 | — | |
| Pointer-GeneratorDataset Version=Non-anonymized, Encoder Architecture=LSTM, Reference Source=See et al. (2017)2021.12 | 36.44 | 15.66 | 33.42 | — | |
| Pointer-Generator (our implementation)beam size=52018.08 | 36.25 | 16.17 | 33.41 | — | |
| Pre-train Encoder onlyModel layers=12, Architecture=Encoder-Decoder2019.05 | 36.05 | 15.48 | 33.48 | — | |
| Nallapati (2016)Dataset Version=anonymous2018.04 | 35.46 | 13.3 | 32.65 | — | |
| Abstractive Model*Dataset Version=Anonymized, Reference Source=Nallapati et al. (2016)2021.12 | 35.46 | 13.3 | 32.65 | — | |
| GPT-2Model layers=48, Zero-shot=true2019.05 | 29.34 | 8.27 | 26.58 | — | |
| Pre-train Decoder onlyModel layers=12, Architecture=Encoder-Decoder2019.05 | 27.48 | 6.87 | 25.4 | — |