Natural Language Understanding on SuperGLUE
91.3SGLUE ScoreVega v2
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Vega v2Rank=12022.12 | 91.3 | 90.5 | 98.6 | 99.2 | 99.4 | 88.2 | 62.4 | 94.4 | 93.9 | 96 | 77.4 | 98.6 | — | — | — | |
| SPoTModel size=XXL2021.10 | 91.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ST-MoE-32BRank=2, Parameters=32B2022.12 | 91.2 | 92.4 | 96.9 | 98 | 99.2 | 89.6 | 65.8 | 95.1 | 94.4 | 93.5 | 77.7 | 96.6 | — | — | — | |
| Turing NLR v5Rank=32022.12 | 90.9 | 92 | 95.9 | 97.6 | 98.2 | 88.4 | 63 | 96.4 | 95.9 | 94.1 | 77.1 | 97.3 | — | — | — | |
| ERNIE 3.0Rank=42022.12 | 90.6 | 91 | 98.6 | 99.2 | 97.4 | 88.6 | 63.2 | 94.7 | 94.2 | 92.6 | 77.4 | 97.3 | — | — | — | |
| PaLM 540BRank=5, Parameters=540B2022.12 | 90.4 | 91.9 | 94.4 | 96 | 99 | 88.7 | 63.6 | 94.2 | 93.3 | 94.1 | 77.4 | 95.9 | — | — | — | |
| T5+UKGRank=6, Model Type=Single Model2022.12 | 90.4 | 91.4 | 95.8 | 97.6 | 98 | 88.3 | 63 | 94.2 | 93.5 | 93 | 77.9 | 96.6 | — | — | — | |
| DeBERTa/ TuringNLRv4Rank=72022.12 | 90.3 | 90.3 | 95.7 | 97.6 | 98.4 | 88.2 | 63.7 | 94.5 | 94.1 | 93.2 | 77.5 | 95.9 | — | — | — | |
| MULTI-TASKMODELTUNINGModel size=XXL2021.10 | 90.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MODELTUNINGModel size=XXL2021.10 | 89.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LOMOBackbone=LLaMA, Params=65B, Training examples=1,0002023.06 | 89.9 | 90.7 | — | — | 97 | — | — | — | — | 93.9 | 75.4 | 92.3 | — | 89.9 | — | |
| SuperGLUE Human BaselinesRank=82022.12 | 89.8 | 89 | 95.8 | 98.9 | 100 | 81.8 | 51.9 | 91.7 | 91.3 | 93.6 | 80 | 100 | — | — | — | |
| T5Rank=92022.12 | 89.3 | 91.2 | 93.9 | 96.8 | 94.8 | 88.1 | 63.3 | 94.1 | 93.4 | 92.5 | 76.9 | 93.8 | — | — | — | |
| Frozen T5 1.1 + SPOTRank=102022.12 | 89.2 | 91.1 | 95.8 | 97.6 | 95.6 | 87.9 | 61.9 | 93.3 | 92.4 | 92.9 | 75.8 | 93.8 | — | — | — | |
| LoRABackbone=LLaMA, Params=65B, Training examples=1,0002023.06 | 89 | 90.9 | — | — | 97 | — | — | — | — | 93.1 | 74.5 | 88.5 | — | 90 | — | |
| PROMPTTUNINGModel size=XXL2021.10 | 88.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPoTModel size=XL2021.10 | 88.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MULTI-TASKMODELTUNINGModel size=XL2021.10 | 88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LOMOBackbone=LLaMA, Params=30B, Training examples=1,0002023.06 | 87.1 | 89.3 | — | — | 93 | — | — | — | — | 92.8 | 74.1 | 85.6 | — | 87.9 | — | |
| LoRABackbone=LLaMA, Params=30B, Training examples=1,0002023.06 | 86.4 | 89.7 | — | — | 93 | — | — | — | — | 91 | 74 | 83.7 | — | 87 | — | |
| MULTI-TASKMODELTUNINGModel size=LARGE2021.10 | 84.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LOMOBackbone=LLaMA, Params=13B, Training examples=1,0002023.06 | 84.2 | 87.3 | — | — | 93 | — | — | — | — | 89.9 | 74.3 | 75 | — | 85.7 | — | |
| MODELTUNINGModel size=XL2021.10 | 83.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPoTModel size=LARGE2021.10 | 82.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRA#T.P.=0.8M, #F.P.=100%2024.06 | 82 | 84.1 | — | — | — | — | — | — | — | — | 72.7 | — | 88.3 | 82.7 | — | |
| Adapter#T.P.=0.8M, #F.P.=100%2024.06 | 81.8 | 84.3 | — | — | — | — | — | — | — | — | 71 | — | 89.5 | 82.4 | — | |
| LoRABackbone=LLaMA, Params=13B, Training examples=1,0002023.06 | 81.4 | 87.1 | — | — | 92 | — | — | — | — | 89.9 | 69.9 | 63.5 | — | 86.1 | — | |
| MODELTUNINGModel size=LARGE2021.10 | 81.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LOMOBackbone=LLaMA, Params=7B, Training examples=1,0002023.06 | 80.8 | 87.5 | — | — | 89 | — | — | — | — | 86.6 | 71.2 | 66.4 | — | 84 | — | |
| Light-PEFT (LoRA)#T.P.=0.3M, #F.P.=76%2024.06 | 79.5 | 83.3 | — | — | — | — | — | — | — | — | 70.2 | — | 86.6 | 78 | — | |
| PROMPTTUNINGModel size=XL2021.10 | 79.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MULTI-TASKMODELTUNINGModel size=BASE2021.10 | 79.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=LLaMA, Params=7B, Training examples=1,0002023.06 | 78.8 | 85.2 | — | — | 87 | — | — | — | — | 85.9 | 65.5 | 64.4 | — | 84.8 | — | |
| Light-PEFT (Adapter)#T.P.=0.3M, #F.P.=76%2024.06 | 78.3 | 81.2 | — | — | — | — | — | — | — | — | 70.1 | — | 86 | 76 | — | |
| Primer-EZ DecoderParams=224M2021.09 | 76.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SCALEARN++Backbone=RoBERTa-base, Learning Paradigm=Two-stage Multi-Task Learning (MTL)2023.10 | 75.74 | 76.44 | — | — | 75.2 | — | — | — | — | 79.03 | 67.13 | 62.26 | 80.13 | 72.71 | 93.04 | |
| Transformer++Params=224M2021.09 | 75.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SCALEARNBackbone=RoBERTa-base, Learning Paradigm=Two-stage Multi-Task Learning (MTL)2023.10 | 75.55 | 77.27 | — | — | 74.8 | — | — | — | — | 78.88 | 66.35 | 63.46 | 79.52 | 73.22 | 90.89 | |
| SCALEARNUNIFORM++Backbone=RoBERTa-base, Learning Paradigm=Two-stage Multi-Task Learning (MTL)2023.10 | 74.93 | 76.13 | — | — | 74 | — | — | — | — | 74.84 | 67.87 | 63.46 | 79.79 | 71.75 | 91.61 | |
| PROMPTTUNINGModel size=LARGE2021.10 | 74.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SCALEARNUNIFORMBackbone=RoBERTa-base, Learning Paradigm=Two-stage Multi-Task Learning (MTL)2023.10 | 74.2 | 76.06 | — | — | 71.2 | — | — | — | — | 75.31 | 67.37 | 62.5 | 80.13 | 71.91 | 89.11 | |
| MODELTUNINGModel size=BASE2021.10 | 73.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PROPETLBackbone=RoBERTa-base, Learning Paradigm=Single Task Learning (STL)2023.10 | 73.69 | 76.58 | — | — | 70.6 | — | — | — | — | 74.19 | 66.6 | 63.46 | 80.29 | 73.07 | 84.46 | |
| Transformer+GeLUParams=223M2021.09 | 73.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADAPTERFUSIONBackbone=RoBERTa-base, Learning Paradigm=Two-stage Multi-Task Learning (MTL)2023.10 | 73.6 | 76.72 | — | — | 73.1 | — | — | — | — | 76.03 | 66.57 | 63.46 | 78.82 | 71.79 | 82.32 | |
| SPoTModel size=BASE2021.10 | 73.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADAPTERBackbone=RoBERTa-base, Learning Paradigm=Single Task Learning (STL)2023.10 | 73.19 | 76.71 | — | — | 70.2 | — | — | — | — | 72.89 | 65.58 | 63.46 | 79.02 | 72.84 | 84.82 | |
| COMPACTER++Backbone=RoBERTa-base, Learning Paradigm=Single Task Learning (STL)2023.10 | 72.74 | 75.88 | — | — | 68.3 | — | — | — | — | 72.02 | 66.46 | 63.46 | 77.69 | 70.44 | 87.68 | |
| LORABackbone=RoBERTa-base, Learning Paradigm=Single Task Learning (STL)2023.10 | 72.72 | 76.58 | — | — | 68.2 | — | — | — | — | 70.4 | 65.14 | 63.46 | 79.6 | 71.96 | 86.43 | |
| (IA)3Backbone=RoBERTa-base, Learning Paradigm=Single Task Learning (STL)2023.10 | 72.69 | 76.31 | — | — | 69.3 | — | — | — | — | 72.56 | 67.07 | 63.35 | 75.27 | 70.32 | 87.32 | |
| FINETUNEBackbone=RoBERTa-base, Learning Paradigm=Single Task Learning (STL)2023.10 | 71.74 | 76.8 | — | — | 68.6 | — | — | — | — | 73.47 | 66.38 | 63.46 | 71.61 | 71.64 | 81.96 | |
| FINETUNE-MBackbone=RoBERTa-base, Learning Paradigm=Joint Multi-Task Learning (MTL)2023.10 | 71.26 | 76.39 | — | — | 74.33 | — | — | — | — | 74.85 | 52.19 | 63.46 | 72.21 | 72.11 | 84.52 | |
| Vanilla TransformerParams=223M2021.09 | 70.97 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADAPTER-MBackbone=RoBERTa-base, Learning Paradigm=Joint Multi-Task Learning (MTL)2023.10 | 70.83 | 75.32 | — | — | 71.67 | — | — | — | — | 76.53 | 51.99 | 59.94 | 72.43 | 72.46 | 86.31 | |
| PROPETL-MBackbone=RoBERTa-base, Learning Paradigm=Joint Multi-Task Learning (MTL)2023.10 | 69.91 | 73.91 | — | — | 74 | — | — | — | — | 73.65 | 50.73 | 59.62 | 73.14 | 72.07 | 82.14 | |
| MULTI-TASKMODELTUNINGModel size=SMALL2021.10 | 64.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPoTModel size=SMALL2021.10 | 64.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PROMPTTUNINGModel size=BASE2021.10 | 63.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MODELTUNINGModel size=SMALL2021.10 | 62.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Zero-shotBackbone=LLaMA, Params=65B2023.06 | 61.3 | 73.6 | — | — | 91 | — | — | — | — | 59.6 | 51.3 | 44.2 | — | 48.3 | — | |
| PROMPTTUNINGModel size=SMALL2021.10 | 59.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HYPERFORMERBackbone=RoBERTa-base, Learning Paradigm=Joint Multi-Task Learning (MTL)2023.10 | 58.81 | 74.01 | — | — | 55.5 | — | — | — | — | 61.73 | 55.49 | 52.88 | 65.93 | 33.54 | 71.43 | |
| ADAPTERSOUPBackbone=RoBERTa-base, Learning Paradigm=Two-stage Multi-Task Learning (MTL)2023.10 | 58.73 | 68.84 | — | — | 52.4 | — | — | — | — | 57.83 | 58.53 | 63.46 | 64.26 | 33.62 | 70.89 | |
| Zero-shotBackbone=LLaMA, Params=30B2023.06 | 58.4 | 74.6 | — | — | 89 | — | — | — | — | 53.4 | 50 | 36.5 | — | 46.9 | — | |
| Zero-shotBackbone=LLaMA, Params=13B2023.06 | 57.2 | 65 | — | — | 88 | — | — | — | — | 60.6 | 49.5 | 36.5 | — | 43.4 | — | |
| Zero-shotBackbone=LLaMA, Params=7B2023.06 | 56.2 | 66.5 | — | — | 85 | — | — | — | — | 57 | 49.7 | 36.5 | — | 42.3 | — | |
| Hybrid H3Parameters=2.7B, Few-shot=3-shot, Scoring=logit scoring2022.12 | 55.5 | 60.6 | — | 46.4 | 77 | 59.4 | — | — | 71.1 | 47.3 | 45.6 | 36.5 | — | — | — | |
| Hybrid H3Parameters=125M, Few-shot=3-shot, Scoring=logit scoring2022.12 | 53.7 | 56.1 | — | 51.8 | 67 | 48.9 | — | — | 55 | 58.1 | 49.1 | 43.3 | — | — | — | |
| Hybrid H3Parameters=1.3B, Few-shot=3-shot, Scoring=logit scoring2022.12 | 53 | 56.9 | — | 23.2 | 76 | 59.3 | — | — | 67.6 | 55.2 | 49.2 | 36.5 | — | — | — | |
| OPTParameters=2.7B, Few-shot=3-shot, Scoring=logit scoring2022.12 | 53 | 62 | — | 17.9 | 71 | 59.2 | — | — | 66 | 53.4 | 50.5 | 44.2 | — | — | — | |
| PROMPTDESIGN (GPT-3)Model size=XXL2021.10 | 52.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid H3Parameters=355M, Few-shot=3-shot, Scoring=logit scoring2022.12 | 52.6 | 60.5 | — | 28.6 | 69 | 59.7 | — | — | 62.3 | 50.5 | 47.5 | 42.3 | — | — | — | |
| GPT-NeoParameters=2.7B, Few-shot=3-shot, Scoring=logit scoring2022.12 | 51.9 | 56 | — | 21.4 | 68 | 57 | — | — | 60 | 51.6 | 51.9 | 49 | — | — | — | |
| GPT-NeoParameters=1.3B, Few-shot=3-shot, Scoring=logit scoring2022.12 | 51.2 | 61.2 | — | 32.1 | 67 | 59.9 | — | — | 55.7 | 47.3 | 50.6 | 35.6 | — | — | — | |
| OPTParameters=350M, Few-shot=3-shot, Scoring=logit scoring2022.12 | 50.1 | 61.7 | — | 44.6 | 60 | 49.8 | — | — | 51.4 | 45.8 | 50 | 37.5 | — | — | — | |
| OPTParameters=1.3B, Few-shot=3-shot, Scoring=logit scoring2022.12 | 49.4 | 38.3 | — | 16.1 | 70 | 59.9 | — | — | 62.1 | 53.4 | 51.1 | 44.2 | — | — | — | |
| OPTParameters=125M, Few-shot=3-shot, Scoring=logit scoring2022.12 | 47.9 | 41.9 | — | 44.6 | 60 | 57.9 | — | — | 44.9 | 47.3 | 50.2 | 36.5 | — | — | — | |
| PROMPTDESIGN (GPT-3)Model size=XL2021.10 | 47.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-NeoParameters=125M, Few-shot=3-shot, Scoring=logit scoring2022.12 | 47.2 | 62.1 | — | 17.9 | 60 | 56.3 | — | — | 39.6 | 53.1 | 50 | 38.5 | — | — | — | |
| HYPERFORMER++Backbone=RoBERTa-base, Learning Paradigm=Joint Multi-Task Learning (MTL)2023.10 | 46.55 | 62.17 | — | — | 54.33 | — | — | — | — | 49.09 | 50 | 63.46 | 24.5 | 19.47 | 49.4 | |
| GPT-2 mediumParameters=355M, Few-shot=3-shot, Scoring=logit scoring2022.12 | 45.6 | 58.8 | — | 8.9 | 65 | 43.5 | — | — | 53.3 | 48 | 50.5 | 36.5 | — | — | — | |
| PROMPTDESIGN (GPT-3)Model size=LARGE2021.10 | 45.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PROMPTDESIGN (GPT-3)Model size=BASE2021.10 | 43.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PROMPTDESIGN (GPT-3)Model size=SMALL2021.10 | 40.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AGZOBackbone=Qwen3-0.6b, Fine-tuning Protocol=Zeroth-order fine-tuning2026.01 | — | 72.4 | — | 89.2 | 74 | — | — | — | — | 77.2 | 59.5 | — | — | 75.6 | — | |
| FOBackbone=Qwen3-0.6b, Fine-tuning Protocol=First-order fine-tuning2026.01 | — | 76.8 | — | 94.6 | 73 | — | — | — | — | 80.8 | 67.5 | — | — | 82.6 | — | |
| ICLBackbone=Qwen3-0.6b, Fine-tuning Protocol=In-context learning2026.01 | — | 70 | — | 57 | 62 | — | — | — | — | 72.2 | 52.3 | — | — | 67.3 | — | |
| LOZOBackbone=Qwen3-0.6b, Fine-tuning Protocol=Zeroth-order fine-tuning2026.01 | — | 72.4 | — | 76 | 69 | — | — | — | — | 74.3 | 57.5 | — | — | 73.7 | — | |
| MeZOModel=LLaMA-7b2024.02 | — | 75.9 | — | — | — | — | — | — | — | 71.7 | 61.4 | — | — | — | — | |
| MeZOModel=LLaMA-30b2024.02 | — | 83.8 | — | — | — | — | — | — | — | 76.9 | 63.3 | — | — | — | — | |
| MEZOBackbone=Qwen3-0.6b, Fine-tuning Protocol=Zeroth-order fine-tuning2026.01 | — | 73 | — | 80.3 | 68 | — | — | — | — | 73.2 | 57.3 | — | — | 73.4 | — | |
| S-MeZOModel=LLaMA-7b2024.02 | — | 80.9 | — | — | — | — | — | — | — | 80.7 | 64.9 | — | — | — | — | |
| S-MeZOModel=LLaMA-30b2024.02 | — | 85.7 | — | — | — | — | — | — | — | 82.1 | 67.3 | — | — | — | — | |
| ZeroBackbone=Qwen3-0.6b, Fine-tuning Protocol=Zero-shot prompting2026.01 | — | 64.6 | — | 41 | 57 | — | — | — | — | 59.9 | 49.8 | — | — | 51.8 | — |