Abstractive Summarization on WikiHow
22.12ROUGE-2PEGASUS_LARGE
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| PEGASUS_LARGEPre-training strategy=mixed, stochastic2019.12 | 22.12 | 46.39 | 38.41 | — | — | |
| UNISUMMFew-shot samples=1002022.11 | 11.73 | 34.9 | 25.7 | — | — | |
| MultiBARTFew-shot samples=1002022.11 | 11.31 | 34.1 | 25.03 | — | — | |
| PEGASUSFew-shot samples=1002022.11 | 10.86 | 33.21 | 24.41 | — | — | |
| BART-PTFew-shot samples=1002022.11 | 10.42 | 32.35 | 23.23 | — | — | |
| UNISUMMFew-shot samples=102022.11 | 9.35 | 30.87 | 21.72 | — | — | |
| GPT-3.5 (text-davinci-002)Few-shot samples=12022.11 | 8.56 | — | — | — | — | |
| MultiBARTFew-shot samples=102022.11 | 7.99 | 27.64 | 19.91 | — | — | |
| PEGASUSFew-shot samples=102022.11 | 7.8 | 27.74 | 19.61 | — | — | |
| BART-PTFew-shot samples=102022.11 | 7.45 | 25.31 | 19.02 | — | — | |
| SummScore - Random-3Backbone=ChatGPT, Candidate Selection=SummScore - Random-32022.12 | 7.26 | 28.091 | 24.42 | 86.39 | 1.27 | |
| SummScore - LEAD-3Backbone=ChatGPT, Candidate Selection=SummScore - LEAD-32022.12 | 7.16 | 28.22 | 24.39 | 86.41 | 7.27 | |
| SummScore - Salient-R1Backbone=ChatGPT, Candidate Selection=SummScore - Salient-R12022.12 | 7.08 | 27.89 | 24.08 | 86.25 | 5.98 | |
| SummScore - Salient-RLBackbone=ChatGPT, Candidate Selection=SummScore - Salient-RL2022.12 | 7.08 | 28.011 | 24.211 | 86.21 | 6.46 | |
| SummScore - Salient-R2Backbone=ChatGPT, Candidate Selection=SummScore - Salient-R22022.12 | 7.04 | 27.931 | 24.141 | 86.24 | 6.09 | |
| BART-FTFew-shot samples=1002022.11 | 6.91 | 26.46 | 18.83 | — | — | |
| SummScore - Salient-R1Backbone=PEGASUS, Candidate Selection=SummScore - Salient-R12022.12 | 6.32 | 26.37 | 18.81 | 84.92 | 4.25 | |
| SummScore - Salient-RLBackbone=PEGASUS, Candidate Selection=SummScore - Salient-RL2022.12 | 6.32 | 26.37 | 18.81 | 84.92 | 4.31 | |
| FirstBackbone=ChatGPT, Candidate Selection=First2022.12 | 6.32 | 26.64 | 22.75 | 86.12 | — | |
| SummScore - Salient-R2Backbone=PEGASUS, Candidate Selection=SummScore - Salient-R22022.12 | 6.3 | 26.4 | 18.83 | 84.92 | 4.37 | |
| SummScore - Random-3Backbone=PEGASUS, Candidate Selection=SummScore - Random-32022.12 | 6.28 | 26.297 | 18.78 | 84.98 | 3.89 | |
| SummScore - LEAD-3Backbone=PEGASUS, Candidate Selection=SummScore - LEAD-32022.12 | 6.19 | 26.17 | 18.69 | 84.96 | 3.16 | |
| Top beamBackbone=PEGASUS, Candidate Selection=Top beam2022.12 | 6 | 25.39 | 18.09 | 84.82 | — | |
| Random beamBackbone=PEGASUS, Candidate Selection=Random beam2022.12 | 5.91 | 25.49 | 17.99 | 84.98 | -0.38 | |
| RandomBackbone=ChatGPT, Candidate Selection=Random2022.12 | 5.8 | 25.27 | 21.78 | 85.31 | -5.26 | |
| BART-FTFew-shot samples=102022.11 | 2.37 | 17.09 | 12.01 | — | — |