Summarization on XSum (test)
60.61ROUGE-2ORACLE
Evaluation results
| Method | Links | |||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ORACLEContext=PEGFAME baseline2021.05 | 60.61 | 82.39 | 69.19 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ORACLEContext=ROBFAME baseline2021.05 | 42.22 | 72.22 | 53.89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OracleSetting=beam-sim2026.05 | 32.27 | 54.1 | 47.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 81.15 | 63.31 | 49.79 | -5.26 | 13.82 | 93.72 | 87.59 | 58.15 | 98.65 | |
| OracleSetting=epsilon2026.05 | 31.35 | 53.82 | 46.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 81.63 | 63.7 | 43.6 | -5.32 | 17.78 | 94.11 | 86.48 | 51.1 | 99.21 | |
| Max (Oracle Selection)Strategy=Candidate selection based on highest ROUGE2021.06 | 28.28 | 52.45 | 43.36 | 72.56 | 62.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ST-MoENumber of parameters=268B2022.12 | 27.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Zoph et al.status=previous state-of-the-art2022.05 | 27.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ST-MoENumber of Parameters=268B2022.08 | 27.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SLiCNumber of parameters=2B2022.12 | 27.09 | 49.77 | 42.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BRIO_jointMemory=Self2023.05 | 26.7 | 50.3 | 41.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UL20Bmode=supervised finetuning2022.05 | 26.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UL2Number of Parameters=20B2022.08 | 26.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BRIO_jointMemory=Retrieval2023.05 | 26.5 | 49.5 | 41.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BRIO_dual*Memory=Self2023.05 | 26.2 | 49.2 | 40.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BRIO_dual*Memory=Retrieval2023.05 | 26.1 | 48.6 | 40.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| REINA (PG)Memory=Retrieval2023.05 | 26 | 48.2 | 40.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MoCaNumber of parameters=568M, Backbone=PEGASUS2022.12 | 25.91 | 49.32 | 41.47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BRIOMemory=None2023.05 | 25.6 | 49.1 | 40.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BRIO-MulBackbone=PEGASUS, Training Loss=Multi-task Loss, Training Mode=Fine-tuned2022.03 | 25.59 | 49.07 | 40.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BRIONumber of parameters=568M, Backbone=PEGASUS2022.12 | 25.59 | 49.07 | 40.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BRIO-CtrBackbone=PEGASUS, Training Loss=Contrastive Loss Only, Training Mode=Fine-tuned2022.03 | 25.13 | 48.13 | 39.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Our Model2021.09 | 25.08 | 49.57 | 41.81 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FROSTDecoding=Single2022.03 | 25.06 | 47.8 | 39.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FROSTdecoding_type=Single2022.03 | 25.06 | 47.8 | 39.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUSDecoding=Single2022.03 | 24.87 | 47.56 | 39.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUSdecoding_type=Single2022.03 | 24.87 | 47.56 | 39.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Pegasus (Reported)Model Size=Large2020.07 | 24.83 | 47.6 | 39.64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUS2021.04 | 24.83 | 47.6 | 39.64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Z-Code++Number of Parameters=800M2022.08 | 24.7 | 47.7 | 39.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUSBackbone=PEGASUS, Evaluation Source=Current Paper Evaluation Script2022.03 | 24.69 | 47.46 | 39.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ConSumBackbone=BART, Evaluation Source=Original Paper2022.03 | 24.67 | 47.34 | 39.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Z-Code++Number of parameters=710M2022.12 | 24.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUS-LargeNumber of Parameters=500M2022.08 | 24.6 | 47.2 | 39.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUS_LARGENumber of Parameters=470M2022.08 | 24.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Z-Code++_LARGENumber of Parameters=710M2022.08 | 24.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUSMemory=None2023.05 | 24.6 | 47.2 | 39.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimCLSBase Model=Pegasus2021.06 | 24.57 | 47.61 | 39.44 | 69.81 | 61.48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimCLSBackbone=BART, Evaluation Model=RoBERTa, Evaluation Source=Original Paper2022.03 | 24.57 | 47.61 | 39.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUSLARGEModel Size=Large, Pre-training Corpus=HugeNews2019.12 | 24.56 | 47.21 | 39.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUSSource=Zhang et al., 20192021.05 | 24.56 | 47.21 | 39.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Pegasus2021.06 | 24.56 | 47.21 | 39.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUSEvaluation Source=Original Paper2022.03 | 24.56 | 47.21 | 39.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUS fine-tunedφ%=1002021.10 | 24.56 | 47.21 | 39.25 | — | — | 49 | 52 | 49 | 49 | 49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUSEvaluation Protocol=Fully-supervised2022.05 | 24.56 | 47.21 | 39.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUSNumber of parameters=568M, Backbone=PEGASUS2022.12 | 24.56 | 47.21 | 39.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUS2021.09 | 24.56 | 47.21 | 39.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Origin (Pegasus baseline)Base Model=Pegasus2021.06 | 24.53 | 47.1 | 39.23 | 69.48 | 61.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FinetunedNumber of parameters=568M, Backbone=PEGASUS, Protocol=supervised fine-tuning2022.12 | 24.53 | 47.14 | 39.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Pegasus (Re-eval)Model Size=Large2020.07 | 24.31 | 47.37 | 39.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| REINA (L)Memory=Retrieval2023.05 | 24.1 | 46.5 | 38.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BIGBIRD-PegasusModel Size=Large2020.07 | 24.05 | 47.12 | 38.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BigBirdSource sequence length=10242021.04 | 24.05 | 47.12 | 38.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Random (Oracle Selection)Strategy=Candidate selection based on random ROUGE score2021.06 | 23.64 | 46.72 | 38.55 | 69.3 | 61.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLMBackbone=RoBERTa2021.03 | 23.5 | 45.5 | 37.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HAT-BARTSource sequence length=10242021.04 | 22.79 | 45.92 | 37.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGFAME2021.05 | 22.75 | 45.31 | 37.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FAMEDecoding=Single2022.03 | 22.75 | 45.31 | 37.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FAMEdecoding_type=Single2022.03 | 22.75 | 45.31 | 37.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BART + TA2021.09 | 22.68 | 45.76 | 38.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GOLD-sBackbone=BART, Evaluation Source=Original Paper2022.03 | 22.58 | 45.85 | 37.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SeqCoBackbone=BART, Evaluation Source=Original Paper2022.03 | 22.41 | 45.65 | 37.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BARTSize=Large2021.03 | 22.3 | 45.1 | 37.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BART-LargeNumber of Parameters=400M2022.08 | 22.3 | 45.1 | 37.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BART2019.10 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BART_LARGE#Param=400M, Model size=LARGE2020.02 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Previous SOTA2019.12 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BARTGroup=Prior Art2020.07 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FINE-TUNE2021.01 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BART2021.05 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BART2021.06 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BARTDecoding=Single2022.03 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BARTBackbone=BART, Evaluation Source=Original Paper2022.03 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BARTLARGE fine-tunedφ%=1002021.10 | 22.27 | 45.14 | 37.25 | — | — | 49 | 50 | 50 | 52 | 49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BARTEvaluation Protocol=Fully-supervised2022.05 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BARTNumber of parameters=400M, Backbone=BART2022.12 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BART2021.04 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BART2021.09 | 22.27 | 45.14 | 37.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUSSource=ours2021.05 | 22.26 | 44.85 | 37.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GOLD-pBackbone=BART, Evaluation Source=Original Paper2022.03 | 22.26 | 45.75 | 37.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Composition (FROST)Decoding=Diverse (Composition)2022.03 | 22.24 | 45.12 | 36.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FROSTdecoding_type=Diverse, sampling_strategy=Composition2022.03 | 22.24 | 45.12 | 36.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-BARTSource sequence length=10242021.04 | 22.19 | 45.26 | 37.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGASUSLARGEModel Size=Large, Pre-training Corpus=C42019.12 | 22.06 | 45.2 | 36.99 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PEGFAMEAblation=w/o Eq. (3)2021.05 | 22 | 44.54 | 36.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5-LargeNumber of Parameters=800M2022.08 | 22 | 44.3 | 36.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GSum2021.06 | 21.89 | 45.4 | 36.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GSumDecoding=Single2022.03 | 21.89 | 45.4 | 36.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GSumdecoding_type=Single2022.03 | 21.89 | 45.4 | 36.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GSumBackbone=BART, Evaluation Source=Original Paper2022.03 | 21.89 | 45.4 | 36.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FROST++Decoding=Single2022.03 | 21.58 | 44.94 | 37.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FROST++decoding_type=Single2022.03 | 21.58 | 44.94 | 37.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PaLMNumber of Parameters=540B2022.08 | 21.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UNILMv2_BASE#Param=110M, Model size=BASE2020.02 | 21.11 | 44 | 36.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UniLMv2Size=Base2021.03 | 21.1 | 44 | 36.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| REINA (B)Memory=Retrieval2023.05 | 21 | 43.2 | 35.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PREFIXparameter percentage=2%2021.01 | 20.93 | 43.8 | 36.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CONTROL PREFIXES (A1, A2)φ%=2.82021.10 | 20.84 | 43.81 | 35.81 | — | — | 51 | 53 | 51 | 53 | 49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BART (Full Precision)W-E-A (#bits)=32-32-32, E-D (#layers)=6-6, Size (MB)=5312022.03 | 20.79 | 43.84 | 35.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UNILMv2_BASE#Param=110M, Model size=BASE, relative position bias=false2020.02 | 20.71 | 43.69 | 35.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Prefix-tuningφ%=3.02021.10 | 20.66 | 43.53 | 35.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |