Text Summarization on CNN/Daily Mail (test)
38.42ROUGE-2Top Down Transformer
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Top Down Transformerpooling=OracleAdaPool2022.03 | 38.42 | 63.87 | 59.1 | — | |
| ORACLEModel Type=Reproduced baseline, Input length=1,024 tok.2022.03 | 32.78 | 55.45 | 51.59 | — | |
| ORACLEModel Type=Reproduced baseline, Input length=512 tok.2022.03 | 30.76 | 52.46 | 48.66 | — | |
| BART (Full Precision)W-E-A (#bits)=32-32-32, E-D (#layers)=6-6, Size (MB)=5312022.03 | 22.25 | 44.9 | 42.09 | — | |
| Our Modelbeam size=8, checkpoints=top-3 average2021.09 | 21.95 | 44.52 | 41.39 | — | |
| DQ-BART (Distillation-Aware Quantization)W-E-A (#bits)=8-8-8, E-D (#layers)=6-6, Size (MB)=1372022.03 | 21.92 | 44.66 | 41.86 | — | |
| T5Corpus=750GB2020.01 | 21.55 | 43.52 | 40.69 | — | |
| Fourier-BART-FPModel Size=400M, Further Pre-trained=true2023.05 | 21.55 | 44.76 | 41.34 | — | |
| Fourier-BARTModel Size=400M2023.05 | 21.48 | 44.65 | 41.3 | — | |
| PEGASUS_LARGE (HugeNews)Corpus=3800GB2020.01 | 21.47 | 44.17 | 41.11 | — | |
| PEGASUSModel Type=Abstractive2022.03 | 21.47 | 44.17 | 41.11 | — | |
| PEGASUS2021.09 | 21.47 | 44.17 | 41.11 | — | |
| LongT5Model size=xl, Input length=4k, Attention mechanism=TGlobal2021.12 | 21.4 | 43.94 | 41.28 | — | |
| BART + TAmodule=topic assistant2021.09 | 21.39 | 44.47 | 41.32 | — | |
| Top Down Transformerpooling=AdaPool2022.03 | 21.31 | 44.85 | 41.15 | — | |
| HAT-BARTBackbone=BART2021.12 | 21.31 | 44.48 | 41.52 | — | |
| BARTCorpus=160GB2020.01 | 21.28 | 44.16 | 40.9 | — | |
| BARTstatus=Reported2022.03 | 21.28 | 44.15 | 40.9 | — | |
| BARTModel Type=Abstractive2022.03 | 21.28 | 44.16 | 40.9 | — | |
| BARTModel Size=400M2023.05 | 21.28 | 44.16 | 40.9 | — | |
| BART2021.09 | 21.28 | 44.16 | 40.9 | — | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=8-8-8, E-D (#layers)=6-3, Size (MB)=1102022.03 | 21.25 | 43.99 | 41.24 | — | |
| PEGASUS_LARGE (C4)Corpus=750GB2020.01 | 21.2 | 43.9 | 40.76 | — | |
| ProphetNetCorpus=160GB2020.01 | 21.17 | 44.2 | 41.3 | — | |
| BigBird PEGASUSModel Type=Abstractive2022.03 | 21.11 | 43.84 | 40.74 | — | |
| BigBird-LargeModel Size=Not mentioned2023.05 | 21.11 | 43.84 | 40.74 | — | |
| Top Down Transformerpooling=AvgPool2022.03 | 21.03 | 44.32 | 41.4 | — | |
| MatchSumModel Type=Hybrid, Base Model=ROBERTa-base2022.03 | 20.86 | 44.41 | 40.55 | — | |
| MatchSUM2021.09 | 20.86 | 44.41 | 40.55 | — | |
| BARTstatus=Re-evaluated2022.03 | 20.81 | 43.93 | 40.79 | — | |
| HiStruct+ & MatchSumModel Type=Hybrid, Base Model=ROBERTa-base, Input length=1,024 tok., MatchSum Component=RoBERTa-base2022.03 | 20.73 | 44.31 | 40.47 | — | |
| ST-MOE-LModel Size=770M2023.05 | 20.7 | — | — | — | |
| MatchSumModel Type=Hybrid, Base Model=BERT-base2022.03 | 20.62 | 44.22 | 40.38 | — | |
| BertSUM + TAmodule=topic assistant2021.09 | 20.58 | 43.06 | 39.67 | — | |
| HiStruct+Model Type=Extractive, Base Model=ROBERTa-base, Input length=1,024 tok.2022.03 | 20.54 | 43.65 | 40.03 | — | |
| TransformerETSModel Type=Reproduced baseline, Base Model=ROBERTa-base, Input length=1,024 tok.2022.03 | 20.53 | 43.62 | 39.99 | — | |
| LongT5Model size=large, Input length=4k, Attention mechanism=TGlobal2021.12 | 20.51 | 42.49 | 40.18 | — | |
| HiStruct+Model Type=Extractive, Base Model=BERT-large, Input length=512 tok.2022.03 | 20.4 | 43.49 | 39.9 | — | |
| TransformerETSModel Type=Reproduced baseline, Base Model=BERT-large, Input length=512 tok.2022.03 | 20.36 | 43.45 | 39.83 | — | |
| BERTSUMEXTModel Type=Extractive, Base Model=BERT-large2022.03 | 20.34 | 43.85 | 39.9 | — | |
| BertSUM2021.09 | 20.34 | 43.85 | 39.9 | — | |
| HiStruct+Model Type=Extractive, Base Model=BERT-base, Input length=1,024 tok.2022.03 | 20.33 | 43.38 | 39.78 | — | |
| TransformerETSModel Type=Reproduced baseline, Base Model=BERT-base, Input length=1,024 tok.2022.03 | 20.27 | 43.32 | 39.69 | — | |
| BERTSUMEXTModel Type=Extractive, Base Model=BERT-base2022.03 | 20.24 | 43.25 | 39.63 | — | |
| DQ-BART (Distillation-Aware Quantization)W-E-A (#bits)=2-2-8, E-D (#layers)=6-6, Size (MB)=392022.03 | 20.07 | 42.94 | 40.13 | — | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=8-8-8, E-D (#layers)=6-1, Size (MB)=922022.03 | 20.04 | 42.52 | 40.05 | — | |
| HIBERTModel Type=Extractive, Base Model=BERT-large2022.03 | 19.95 | 42.37 | 38.83 | — | |
| HIBERTModel Type=Extractive, Base Model=BERT-base2022.03 | 19.87 | 42.31 | 38.78 | — | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=2-2-8, E-D (#layers)=6-3, Size (MB)=322022.03 | 19.71 | 42.49 | 39.7 | — | |
| Switch TransformerModel Size=223M2023.05 | 19.6 | — | — | — | |
| Two-Stage + RLModel Type=Abstractive (Ours), Mechanism=Refine process with RL objective2019.02 | 19.49 | 41.71 | 38.79 | 33.33 | |
| DCAModel Type=Abstractive2019.02 | 19.47 | 41.69 | 37.92 | 33.11 | |
| BERTSUMABSModel Type=Abstractive2022.03 | 19.39 | 41.72 | 38.76 | — | |
| Two-StageModel Type=Abstractive (Ours), Mechanism=Refine decoder added2019.02 | 19.34 | 41.38 | 38.37 | 33.03 | |
| MASK-LMglobalModel Type=Extractive2019.02 | 19.1 | 41.6 | 37.6 | 32.77 | |
| NeuSUMModel Type=Extractive2019.02 | 19.01 | 41.59 | 37.98 | 32.86 | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=8-8-8, E-D (#layers)=3-1, Size (MB)=722022.03 | 18.75 | 41.18 | 38.58 | — | |
| rnn-ext + RLModel Type=Extractive2019.02 | 18.72 | 41.47 | 37.76 | 32.65 | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=2-2-8, E-D (#layers)=6-1, Size (MB)=272022.03 | 18.72 | 41.14 | 38.66 | — | |
| Bottom-Up SummarizationModel Type=Abstractive2019.02 | 18.68 | 41.22 | 38.34 | 32.75 | |
| BottomUp2021.09 | 18.68 | 41.22 | 38.34 | — | |
| RefreshModel Type=Extractive2019.02 | 18.2 | 40 | 36.6 | 31.6 | |
| VHTM2021.09 | 18.05 | 40.57 | 37.18 | — | |
| inconsistency lossModel Type=Abstractive2019.02 | 17.97 | 40.68 | 37.13 | 31.93 | |
| One-StageModel Type=Abstractive (Ours), Mechanism=Copy-mechanism based on BERT2019.02 | 17.87 | 39.5 | 36.65 | 31.34 | |
| DeepChannelModel Type=Extractive2019.02 | 17.77 | 41.5 | 37.62 | 32.3 | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=2-2-8, E-D (#layers)=3-1, Size (MB)=222022.03 | 17.75 | 40.14 | 37.6 | — | |
| lead-3Model Type=Extractive2019.02 | 17.7 | 40.34 | 36.57 | 31.54 | |
| LEAD-3Model Type=Reproduced baseline2022.03 | 17.39 | 40.33 | 36.56 | — | |
| PointerGenerator+CoverageModel Type=Abstractive2019.02 | 17.28 | 39.53 | 36.38 | 31.06 | |
| PTGEN + Covmechanism=coverage2021.09 | 17.28 | 39.56 | 36.38 | — | |
| DQ-BART (Joint Distillation and Quantization)W-E-A (#bits)=2-2-8, E-D (#layers)=1-1, Size (MB)=192022.03 | 16.73 | 39 | 36.42 | — | |
| SummmaRuNNerModel Type=Extractive2019.02 | 16.2 | 39.6 | 35.3 | 30.37 | |
| ML+RL+intra-attnModel Type=Abstractive2019.02 | 15.82 | 39.87 | 36.9 | 30.87 | |
| DRM2021.09 | 15.75 | 41.16 | 39.08 | — | |
| PTGEN2021.09 | 15.66 | 36.44 | 33.42 | — | |
| BART (Direct Quantization)W-E-A (#bits)=8-8-8, E-D (#layers)=6-6, Size (MB)=1372022.03 | 1.01 | 11.36 | 11.01 | — |