Abstractive Summarization on XSum (ROUGE + BERTScore)
40.67ROUGE-1T5-large
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| T5-largeWarmup=true2025.02 | 40.67 | 18.23 | 33.21 | 91.11 | — | |
| T5-largeWarmup=false2025.02 | 40.21 | 17.84 | 32.75 | 91.05 | — | |
| Oracle2026.06 | 39.1 | — | 32 | 90.8 | — | |
| T5-baseWarmup=true2025.02 | 37.63 | 15.51 | 30.32 | 90.5 | — | |
| T5-baseWarmup=false2025.02 | 37.18 | 15.19 | 29.96 | 90.4 | — | |
| DDPM + QS1.0 + SC0.5Model=TEncDM Emb, q-sampling=tact/T = 1.0, Self-conditioning=p=0.52026.04 | 31.9 | 9.9 | 24.5 | 67.5 | — | |
| FirstBackbone=ChatGPT, Candidate Selection=First2022.12 | 30.53 | 10.2 | 22.2 | 88.77 | — | |
| RandomBackbone=ChatGPT, Candidate Selection=Random2022.12 | 30.53 | 10.2 | 22.2 | 88.77 | 0.48 | |
| DDPM + SC0.5Model=TEncDM Emb, Self-conditioning=p=0.52026.04 | 30.5 | 9.1 | 23.4 | 66.3 | — | |
| DDPM + QS1.0Model=TEncDM Emb, q-sampling=tact/T = 1.02026.04 | 30.1 | 8.4 | 23.1 | 63.4 | — | |
| SummScore - Random-3Backbone=ChatGPT, Candidate Selection=SummScore - Random-32022.12 | 27.98 | 8.45 | 19.64 | 87.94 | -10.49 | |
| SummScore - LEAD-3Backbone=ChatGPT, Candidate Selection=SummScore - LEAD-32022.12 | 27.97 | 8.42 | 19.76 | 88.05 | -10.34 | |
| SummScore - Salient-R1Backbone=ChatGPT, Candidate Selection=SummScore - Salient-R12022.12 | 27.84 | 8.46 | 19.55 | 87.91 | -10.87 | |
| SummScore - Salient-RLBackbone=ChatGPT, Candidate Selection=SummScore - Salient-RL2022.12 | 27.82 | 8.51 | 19.58 | 87.9 | -10.73 | |
| SummScore - Salient-R2Backbone=ChatGPT, Candidate Selection=SummScore - Salient-R22022.12 | 27.79 | 8.47 | 19.57 | 87.9 | -10.87 | |
| GLASSselection rate=50%2026.06 | 27.7 | — | 21.9 | 88.2 | — | |
| DDPMModel=TEncDM Emb2026.04 | 27.2 | 6.7 | 20.7 | 59.3 | — | |
| MLP (RL)selection rate=50%2026.06 | 24.5 | — | 19.6 | 84.9 | — | |
| SummScore - LEAD-3Backbone=PEGASUS, Candidate Selection=SummScore - LEAD-32022.12 | 19.62 | 3.02 | 14.71 | 85.92 | 5.24 | |
| SummScore - Random-3Backbone=PEGASUS, Candidate Selection=SummScore - Random-32022.12 | 19.37 | 2.99 | 14.52 | 85.78 | 3.89 | |
| SummScore - Salient-RLBackbone=PEGASUS, Candidate Selection=SummScore - Salient-RL2022.12 | 19.29 | 2.99 | 14.48 | 85.79 | 3.63 | |
| SummScore - Salient-R2Backbone=PEGASUS, Candidate Selection=SummScore - Salient-R22022.12 | 19.13 | 2.96 | 14.34 | 85.67 | 2.62 | |
| SummScore - Salient-R1Backbone=PEGASUS, Candidate Selection=SummScore - Salient-R12022.12 | 18.96 | 2.88 | 14.19 | 85.65 | 1.52 | |
| Top beamBackbone=PEGASUS, Candidate Selection=Top beam2022.12 | 18.77 | 2.86 | 13.85 | 85.66 | — | |
| Random beamBackbone=PEGASUS, Candidate Selection=Random beam2022.12 | 18.58 | 2.81 | 13.9 | 85.29 | -1.31 |