Abstractive Summarization on XSum
17.9XSum ScoreTransformer++
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Transformer++Params=224M2021.09 | 17.9 | — | — | — | — | |
| Primer-EZ DecoderParams=224M2021.09 | 17.87 | — | — | — | — | |
| Transformer+GeLUParams=223M2021.09 | 17.86 | — | — | — | — | |
| Vanilla TransformerParams=223M2021.09 | 17.78 | — | — | — | — | |
| RWKV-7BBackbone=RWKV-7B, Pruning ratio=0%2024.03 | 16.5 | — | — | — | — | |
| Mamba-2.8BBackbone=Mamba-2.8B, Pruning ratio=0%2024.03 | 15.03 | — | — | — | — | |
| ShortGPTBackbone=Mamba-2.8B, Pruning ratio=10.9%2024.03 | 14.95 | — | — | — | — | |
| ShortGPTBackbone=Mamba-2.8B, Pruning ratio=20.3%2024.03 | 14.71 | — | — | — | — | |
| ShortGPTBackbone=Mamba-2.8B, Pruning ratio=25%2024.03 | 14 | — | — | — | — | |
| ShortGPTBackbone=Mamba-2.8B, Pruning ratio=31.3%2024.03 | 13.77 | — | — | — | — | |
| ShortGPTBackbone=RWKV-7B, Pruning ratio=25%2024.03 | 12.02 | — | — | — | — | |
| ShortGPTBackbone=RWKV-7B, Pruning ratio=28.1%2024.03 | 10.43 | — | — | — | — | |
| ShortGPTBackbone=RWKV-7B, Pruning ratio=9.4%2024.03 | 9.57 | — | — | — | — | |
| ShortGPTBackbone=RWKV-7B, Pruning ratio=18.8%2024.03 | 8.13 | — | — | — | — | |
| ENGINERole=Student2023.07 | — | 39.19 | 16.18 | 31.23 | — | |
| FastGASAnnotation budget (|L|)=1002024.06 | — | — | — | 20 | — | |
| FastGASAnnotation budget (|L|)=182024.06 | — | — | — | 20.26 | — | |
| FastGASModel=GPT-Neo-2.7B, Annotation Budget=182024.06 | — | — | — | — | 20.15 | |
| FastGASModel=OPT-6.7B, Annotation Budget=182024.06 | — | — | — | — | 6.89 | |
| IDEALModel=GPT-Neo-2.7B, Annotation Budget=182024.06 | — | — | — | — | 19.69 | |
| IDEALModel=OPT-6.7B, Annotation Budget=182024.06 | — | — | — | — | 6.13 | |
| JSRole=Student, Divergence=Jensen-Shannon2023.07 | — | 41.65 | 19.22 | 34.03 | — | |
| KLRole=Student, Divergence=Kullback-Leibler2023.07 | — | 41.28 | 18.98 | 33.71 | — | |
| LouvainAnnotation budget (|L|)=182024.06 | — | — | — | 20.23 | — | |
| Non-distill (MLE)Role=Student2023.07 | — | 30 | 10.67 | 24.4 | — | |
| PEGASUS_LARGEPre-training strategy=mixed, stochastic2019.12 | — | 47.6 | 24.83 | 39.64 | — | |
| Pre-distillRole=Student2023.07 | — | 40.58 | 17.79 | 32.55 | — | |
| RKLRole=Student, Divergence=Reverse Kullback-Leibler2023.07 | — | 41.69 | 19.02 | 33.92 | — | |
| SeqKDRole=Student2023.07 | — | 39.13 | 17.53 | 32.34 | — | |
| TeacherRole=Teacher2023.07 | — | 45.12 | 22.26 | 37.18 | — | |
| TVDRole=Student, Divergence=Total Variation2023.07 | — | 41.76 | 19.3 | 34.1 | — | |
| Vote-kModel=GPT-Neo-2.7B, Annotation Budget=182024.06 | — | — | — | — | 19.45 | |
| Vote-kModel=OPT-6.7B, Annotation Budget=182024.06 | — | — | — | — | 6.86 |