Summarization on CNN/DM
56.22ROUGE-1ORACLE
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| ORACLE2022.08 | 56.22 | — | 52.19 | — | — | 33.74 | |
| BRIO-MulEvaluation Source=Original Paper2023.05 | 47.78 | — | 44.57 | — | — | 23.55 | |
| PGAre-ranker=BRIO2023.05 | 47.59 | — | 44.33 | 89.02 | — | 23.81 | |
| BRIO-MulEvaluation Source=Our Evaluation Script2023.05 | 47.5 | — | 44.01 | 89.08 | — | 23.48 | |
| BRIO-CtrEvaluation Source=Original Paper2023.05 | 47.28 | — | 44.15 | — | — | 22.93 | |
| BRIO-Ctrsource=reported in re-ranking papers2023.05 | 47.28 | — | 44.15 | — | — | 22.93 | |
| SummaRerankerEvaluation Source=Original Paper2023.05 | 47.16 | — | 43.87 | 87.74 | — | 22.55 | |
| SummaReRankersource=reported in re-ranking papers2023.05 | 47.16 | — | 43.87 | — | — | 22.55 | |
| BRIO-CtrEvaluation Source=Our Evaluation Script2023.05 | 47.08 | — | 44.06 | 89.03 | — | 23.03 | |
| Diverse Beamre-ranker=BRIO2023.05 | 46.98 | — | 43.85 | 88.95 | — | 22.9 | |
| SimCLSEvaluation Source=Original Paper2023.05 | 46.67 | — | 43.54 | — | — | 22.15 | |
| SimCLSsource=reported in re-ranking papers2023.05 | 46.67 | — | 43.54 | — | — | 22.15 | |
| BalSumEvaluation Source=Our Evaluation Script2023.05 | 46.58 | — | 43.49 | 89.67 | — | 22.33 | |
| Nucleusre-ranker=BRIO2023.05 | 46.57 | — | 43.37 | 88.84 | — | 23.06 | |
| SimCLSEvaluation Source=Our Evaluation Script2023.05 | 46.34 | — | 43.3 | 88.92 | — | 22.07 | |
| Beam Searchre-ranker=BRIO2023.05 | 45.26 | — | 41.87 | 88.52 | — | 22.04 | |
| GRETEL2022.08 | 44.62 | — | 40.69 | — | — | 20.96 | |
| HETFORMER2022.08 | 44.55 | — | 40.37 | — | — | 20.82 | |
| MatchSum2022.08 | 44.41 | — | 40.55 | — | — | 20.86 | |
| BARTEvaluation Source=Original Paper2023.05 | 44.16 | — | 40.9 | — | — | 21.28 | |
| PegasusEvaluation Source=Original Paper2023.05 | 44.16 | — | 41.3 | — | — | 21.56 | |
| BARTEvaluation Source=Our Evaluation Script2023.05 | 44.04 | — | 40.86 | 88.12 | — | 21.06 | |
| Topic-GraphSum2022.08 | 44.02 | — | — | — | — | 20.81 | |
| Top Beamsetup=single candidate2023.05 | 44 | — | 37.42 | 86.38 | — | 21.03 | |
| ETCSum2022.08 | 43.84 | — | 39.77 | — | — | 20.8 | |
| BERTSum+SW2022.08 | 43.78 | — | 39.67 | — | — | 20.65 | |
| BERTSum2022.08 | 43.25 | — | 39.63 | — | — | 20.24 | |
| Longformer-Ext2022.08 | 43 | — | 39.3 | — | — | 20.2 | |
| GBT-EXTSUM2022.08 | 42.93 | — | 39.2 | — | — | 19.81 | |
| HiBERT2022.08 | 42.37 | — | 38.83 | — | — | 19.95 | |
| LEAD2022.08 | 40.11 | — | 36.32 | — | — | 17.54 | |
| SelfGAN_Coop-MCTSGenerator=SelfGAN_Coop-MCTS, Decoder=Coop-MCTS2021.06 | 39 | 12.6 | 37.1 | 23.3 | 15.3 | — | |
| Reformer-Ext2022.08 | 38.85 | — | 35.16 | — | — | 16.46 | |
| ColdGANGenerator=ColdGAN, Decoder=DASglobal2021.06 | 38.8 | 12 | 35.6 | 15.2 | 10.7 | — | |
| SelfGAN_DAS_localGenerator=SelfGAN_DAS_local, Decoder=DASlocal2021.06 | 38.6 | 12.2 | 36.2 | 17.3 | 13.3 | — | |
| SelfGAN_Coop-MCTSGenerator=SelfGAN_Coop-MCTS, Decoder=BeamSearch2021.06 | 38.6 | 12.3 | 36.7 | 20.2 | 12.7 | — | |
| HTLMzero-shot=true, prompting strategy=manually engineered, size hints=true2021.07 | 38.51 | — | 33.89 | — | — | 16.1 | |
| ColdGANGenerator=ColdGAN, Decoder=Coop-MCTS2021.06 | 38.4 | 11.5 | 35.6 | 22.2 | 12.7 | — | |
| SelfGAN_DAS_localGenerator=SelfGAN_DAS_local, Decoder=BeamSearch2021.06 | 38.4 | 12.2 | 36.7 | 15.6 | 11.8 | — | |
| MLEGenerator=MLE, Decoder=DASglobal2021.06 | 38.3 | 11.7 | 36.2 | 11.6 | 9.8 | — | |
| SelfGAN_DAS_localGenerator=SelfGAN_DAS_local, Decoder=DASglobal2021.06 | 38.2 | 12 | 36.6 | 19 | 11.5 | — | |
| MLEGenerator=MLE, Decoder=DASlocal2021.06 | 38.1 | 12 | 35.4 | 16.6 | 11.2 | — | |
| SelfGAN_DAS_localGenerator=SelfGAN_DAS_local, Decoder=Coop-MCTS2021.06 | 38.1 | 11.8 | 37 | 22.6 | 13.3 | — | |
| SelfGAN_DAS_globalGenerator=SelfGAN_DAS_global, Decoder=DASglobal2021.06 | 38 | 12.3 | 36.8 | 16.3 | 11.3 | — | |
| SelfGAN_Coop-MCTSGenerator=SelfGAN_Coop-MCTS, Decoder=DASlocal2021.06 | 38 | 12 | 37 | 24.3 | 13.4 | — | |
| ColdGANGenerator=ColdGAN, Decoder=BeamSearch2021.06 | 37.8 | 11.6 | 36.4 | 14.4 | 9.8 | — | |
| SelfGAN_DAS_globalGenerator=SelfGAN_DAS_global, Decoder=Coop-MCTS2021.06 | 37.7 | 11.6 | 36.6 | 22.4 | 13.6 | — | |
| SelfGAN_DAS_globalGenerator=SelfGAN_DAS_global, Decoder=BeamSearch2021.06 | 37.4 | 12.2 | 35.9 | 14.1 | 11.4 | — | |
| SelfGAN_Coop-MCTSGenerator=SelfGAN_Coop-MCTS, Decoder=DASglobal2021.06 | 37.2 | 11.9 | 35.4 | 22.9 | 12.2 | — | |
| SelfGAN_DAS_globalGenerator=SelfGAN_DAS_global, Decoder=DASlocal2021.06 | 37.1 | 11.5 | 35.1 | 18 | 12.9 | — | |
| MLEGenerator=MLE, Decoder=Coop-MCTS2021.06 | 37 | 11.6 | 35.8 | 19.8 | 11.8 | — | |
| ColdGANGenerator=ColdGAN, Decoder=DASlocal2021.06 | 36.9 | 11.5 | 36.3 | 18.4 | 11.1 | — | |
| MLEGenerator=MLE, Decoder=BeamSearch2021.06 | 36.8 | 11.5 | 34.9 | 8.6 | 8.4 | — | |
| HTLMzero-shot=true, prompting strategy=autoprompting, size hints=true2021.07 | 34.65 | — | 32.15 | — | — | 14.54 | |
| HTLMzero-shot=true, prompting strategy=autoprompting, size hints=false2021.07 | 33.4 | — | 30.1 | — | — | 13.45 | |
| PEGASUSzero-shot=true2021.07 | 32.9 | — | 29.38 | — | — | 13.28 | |
| Base ModelModel Size=13B, Transfer Source=N/A2024.06 | — | — | — | — | — | 8.94 | |
| Full Fine-tuningModel Size=13B, Transfer Source=N/A2024.06 | — | — | — | — | — | 15.5 | |
| Full Fine-tuningModel Size=13B, Transfer Source=1.1B2024.06 | — | — | — | — | — | 14.22 | |
| Full Fine-tuningModel Size=13B, Transfer Source=7B2024.06 | — | — | — | — | — | 15.21 | |
| LoRA TuningModel Size=13B, Transfer Source=N/A2024.06 | — | — | — | — | — | 17.18 | |
| OursModel Size=13B, Transfer Source=1.1B2024.06 | — | — | — | — | — | 14.48 | |
| OursModel Size=13B, Transfer Source=7B2024.06 | — | — | — | — | — | 15.31 | |
| Proxy TuningModel Size=13B, Transfer Source=1.1B2024.06 | — | — | — | — | — | 13.23 | |
| Proxy TuningModel Size=13B, Transfer Source=7B2024.06 | — | — | — | — | — | 14.37 |