Natural Language Understanding on SuperGLUE (test)
92.4BoolQ AccuracyST-MoE-32B
Evaluation results
| Method | Links | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ST-MoE-32BParameters=32B2022.12 | 92.4 | 91.2 | 98 | 96.9 | 99.2 | 93.5 | 77.7 | 96.6 | — | — | 94.4 | 95.1 | 89.6 | 65.8 | — | — | — | — | |
| ST-MoE-32B2022.04 | 92.4 | 91.2 | 96.9 | — | 99.2 | 93.5 | 77.7 | 96.6 | 89.6 | — | 95.1 | — | — | — | — | — | — | — | |
| FLAN-1B# train tasks=1,836, Model Size=1B, Evaluation Protocol=In-Context Learning2024.03 | 92.3 | — | 92.5 | — | 97.8 | 92.1 | 69.9 | 70.7 | 88.3 | — | 38.9 | — | — | — | — | 84.8 | 21.9 | — | |
| FLAN-3B# train tasks=1,836, Model Size=3B, Evaluation Protocol=In-Context Learning2024.03 | 92.1 | — | 92.7 | — | 99.3 | 96 | 79.7 | 80.2 | 90.4 | — | 62.2 | — | — | — | — | 95.3 | 22 | — | |
| Turing NLR v52022.12 | 92 | 90.9 | 97.6 | 95.9 | 98.2 | 94.1 | 77.1 | 97.3 | — | — | 95.9 | 96.4 | 88.4 | 63 | — | — | — | — | |
| PaLM 540BParameters=540B2022.12 | 91.9 | 90.4 | 96 | 94.4 | 99 | 94.1 | 77.4 | 95.9 | — | — | 93.3 | 94.2 | 88.7 | 63.6 | — | — | — | — | |
| PaLM 540BEvaluation Protocol=finetuned2022.04 | 91.9 | 90.4 | 94.4 | — | 99 | 95.9 | 77.4 | 95.9 | 88.7 | — | 94.2 | — | — | — | — | — | — | — | |
| T5-11BParameters=11 billion2019.10 | 91.2 | 88.9 | 96.8 | 93.9 | 94.8 | 92.5 | 76.9 | 93.8 | — | 88.1 | 93.4 | 94.1 | — | 63.3 | — | — | — | — | |
| Fine-tuned SOTAEvaluation protocol=fine-tuned2020.05 | 91 | 89 | 96.9 | 93.9 | 94.8 | 92.5 | 76.1 | 93.8 | 62.3 | 88.2 | 92.5 | 93.3 | — | — | — | — | — | — | |
| ERNIE 3.02022.12 | 91 | 90.6 | 99.2 | 98.6 | 97.4 | 92.6 | 77.4 | 97.3 | — | — | 94.2 | 94.7 | 88.6 | 63.2 | — | — | — | — | |
| Vega v2Parameters=6B2022.12 | 90.5 | 91.3 | 99.2 | 98.6 | 99.4 | 96 | 77.4 | 98.6 | — | — | 93.9 | 94.4 | 88.2 | 62.4 | — | — | — | — | |
| T5-3BParameters=3 billion2019.10 | 89.9 | 86.4 | 94.4 | 90.3 | 92 | 90.7 | 72.1 | 90.4 | — | 86.8 | 90.4 | 91.2 | — | 58.3 | — | — | — | — | |
| SuperGLUE Human Baselines2022.12 | 89 | 89.8 | 98.9 | 95.8 | 100 | 93.6 | 80 | 100 | — | — | 91.3 | 91.7 | 81.8 | 51.9 | — | — | — | — | |
| Previous best2019.10 | 87.1 | 84.6 | 95.2 | 90.5 | 90.6 | 88.2 | 69.9 | 89 | — | 84.4 | 90 | 90.6 | — | 52.5 | — | — | — | — | |
| HiFTModel=Llama 2-7B2026.05 | 86.7 | 83.6 | — | — | 86.5 | 84.7 | 67.6 | — | 81.5 | — | — | — | — | — | — | — | — | 94.7 | |
| AdamModel=Llama 2-7B2026.05 | 86.5 | 83.6 | — | — | 85.6 | 84.5 | 69.5 | — | 80.3 | — | — | — | — | — | — | — | — | 95.4 | |
| APOLLOModel=Llama 2-7B2026.05 | 86.5 | 83.8 | — | — | 86.2 | 84.1 | 70.3 | — | 81.6 | — | — | — | — | — | — | — | — | 94.3 | |
| AdamBackbone=RoBERTa-large2024.04 | 86 | — | — | — | 59 | 87 | 70 | 68 | 76 | — | — | — | — | — | — | — | — | — | |
| ChunkFTModel=Llama 2-7B2026.05 | 85.7 | 84.9 | — | — | 87.6 | 85.7 | 71.2 | — | 82.7 | — | — | — | — | — | — | — | — | 96.5 | |
| BAdamModel=Llama 2-7B2026.05 | 85.6 | 82.5 | — | — | 84.4 | 83.5 | 65.6 | — | 80.3 | — | — | — | — | — | — | — | — | 95.4 | |
| T5-LargeParameters=770 million2019.10 | 85.4 | 82.3 | 94.8 | 91.6 | 83.4 | 87.8 | 69.3 | 86.3 | — | 83.3 | 85.9 | 86.8 | — | 50.7 | — | — | — | — | |
| BAdamBackbone=RoBERTa-large2024.04 | 85 | — | — | — | 69 | 76 | 64 | 65 | 77 | — | — | — | — | — | — | — | — | — | |
| LoRAModel=Llama 2-7B2026.05 | 84.3 | 82.7 | — | — | 86.5 | 83.4 | 67.4 | — | 80.3 | — | — | — | — | — | — | — | — | 94.3 | |
| GaloreModel=Llama 2-7B2026.05 | 84.3 | 83.2 | — | — | 86.5 | 84.3 | 68.4 | — | 81.3 | — | — | — | — | — | — | — | — | 94.3 | |
| T5-BaseParameters=220 million2019.10 | 81.4 | 76.2 | 94 | 86.2 | 71.2 | 81.5 | 68.3 | 80.8 | — | 79.7 | 74.2 | 75 | — | 43.1 | — | — | — | — | |
| S-MeZOModel=Llama 2-7B2026.05 | 81.4 | 80.4 | — | — | 87.5 | 80.4 | 64.5 | — | 73.3 | — | — | — | — | — | — | — | — | 95.5 | |
| LoRABackbone=RoBERTa-large2024.04 | 81 | — | — | — | 56 | 79 | 59 | 62 | 69 | — | — | — | — | — | — | — | — | — | |
| ByT5Model=ByT52023.07 | 79.2 | — | 91.07 | 90.37 | 58 | 81.23 | 70.85 | — | — | — | — | 72.79 | 70.14 | 32 | 72.1 | — | — | — | |
| SuperGLUE Baselines2022.12 | 79 | 71.5 | 90.4 | 84.8 | 73.8 | 79 | 69.6 | 64.4 | — | — | 71.3 | 72 | 70 | 24.1 | — | — | — | — | |
| mT5Model=mT52023.07 | 78.1 | — | 92.86 | 90.53 | 61 | 74.37 | 68.03 | — | — | — | — | 68.26 | 73.03 | 33.68 | 67.22 | — | — | — | |
| Fine-tuned BERT-LargeEvaluation protocol=fine-tuned, Model backbone=BERT-Large2020.05 | 77.4 | 69 | 83.6 | 75.7 | 70.6 | 71.7 | 69.6 | 64.6 | 24.1 | 70 | 71.3 | 72 | — | — | — | — | — | — | |
| GPT-3 Few-ShotEvaluation protocol=few-shot, Number of few-shot examples (K)=322020.05 | 76.4 | 71.8 | 75.6 | 52 | 92 | 69 | 49.4 | 80.1 | 30.5 | 75.4 | 90.2 | 91.1 | — | — | — | — | — | — | |
| T5-SmallParameters=60 million2019.10 | 76.4 | 63.3 | 81.6 | 56.9 | 46 | 73.3 | 66.9 | 70.5 | — | 69.3 | 55.4 | 56.3 | — | 26.3 | — | — | — | — | |
| Best Decoder-only LM2022.04 | 76.4 | 71.8 | 52 | — | 92 | 69 | 49.4 | 80.1 | 75.4 | — | 91.1 | — | — | — | — | — | — | — | |
| MeZOModel=Llama 2-7B2026.05 | 76.2 | 77 | — | — | 86.5 | 72.3 | 61.6 | — | 70.2 | — | — | — | — | — | — | — | — | 95.4 | |
| HyperMoEPLM backbone=Switch Transformer-base-8, Protocol=Fine-tuning, Routing=Top-12024.02 | 73.14 | 64.88 | 69.68 | — | 45 | 67.67 | 65.31 | 56.53 | 67.68 | — | 74.06 | — | — | — | — | — | — | — | |
| MoEPLM backbone=Switch Transformer-base-8, Protocol=Fine-tuning, Routing=Top-12024.02 | 72.69 | 64.04 | 69.64 | — | 45 | 67.15 | 63.63 | 56.58 | 66.38 | — | 71.26 | — | — | — | — | — | — | — | |
| MoE-SharePLM backbone=Switch Transformer-base-8, Protocol=Fine-tuning, Routing=Top-1, Shared MLP=True2024.02 | 72.11 | 64.21 | 67.85 | — | 45 | 67.87 | 65.36 | 56.84 | 66.71 | — | 71.91 | — | — | — | — | — | — | — | |
| Tk-Instruct-3B# train tasks=1,616, Model Size=3B, Evaluation Protocol=In-Context Learning2024.03 | 68.8 | — | 66.3 | — | 73.8 | 62.7 | 50.4 | 51.2 | 59.9 | — | 18.6 | — | — | — | — | 53.5 | 49.9 | — | |
| ICLModel=Llama 2-7B2026.05 | 67.4 | 66.9 | — | — | 84.4 | 55.4 | 53.4 | — | 59.5 | — | — | — | — | — | — | — | — | 81.4 | |
| Zero-ShotModel=Llama 2-7B2026.05 | 65.3 | 61.2 | — | — | 61.5 | 50.4 | 51.3 | — | 57.6 | — | — | — | — | — | — | — | — | 81.3 | |
| Tk-CoAT-1B# train tasks=2, Model Size=1B, Evaluation Protocol=In-Context Learning2024.03 | 63.8 | — | 46.9 | — | 76.1 | 53.7 | 53.5 | 53.6 | 11.4 | — | 17 | — | — | — | — | 50.4 | 52.7 | — | |
| Tk-CoAT-3B# train tasks=2, Model Size=3B, Evaluation Protocol=In-Context Learning2024.03 | 63.6 | — | 60.4 | — | 81.3 | 52 | 56.9 | 53.6 | 56.9 | — | 23.1 | — | — | — | — | 57.9 | 57.2 | — | |
| Tk-Instruct-1B# train tasks=1,616, Model Size=1B, Evaluation Protocol=In-Context Learning2024.03 | 63.4 | — | 46 | — | 76.9 | 55.5 | 53.5 | 49.8 | 62.2 | — | 13.1 | — | — | — | — | 51.9 | 57.2 | — | |
| GPT-Neo-125Mnumber of parameters=125M, zero-shot=true, scoring=logit scoring2022.12 | 62.2 | 50.8 | 41.1 | — | 60 | 53.1 | 53.6 | 36.5 | 59.9 | — | 39.6 | — | — | — | — | — | — | — | |
| GPT-Neo-125Mshots=3-shot, evaluation_method=rank classification, parameters=125M2022.12 | 62.1 | 47.2 | 17.9 | — | 60 | 53.1 | 50 | 38.5 | 56.3 | — | 39.6 | — | — | — | — | — | — | — | |
| Hybrid H3-1.3Bnumber of parameters=1.3B, zero-shot=true, scoring=logit scoring2022.12 | 61.7 | 56.5 | 33.9 | — | 74 | 53.4 | 50.3 | 52.9 | 58.2 | — | 67.8 | — | — | — | — | — | — | — | |
| Hybrid H3-355Mnumber of parameters=355M, zero-shot=true, scoring=logit scoring2022.12 | 61.5 | 54.7 | 41.1 | — | 69 | 55.2 | 51.7 | 37.5 | 59.5 | — | 62.3 | — | — | — | — | — | — | — | |
| GPT-Neo-125MParameters=125M, Shots=3-shot, Mode=generation2022.12 | 61.2 | 47 | 42.9 | — | 68 | 53.1 | 50 | 38.5 | 22.5 | — | 39.7 | — | — | — | — | — | — | — | |
| GPT-Neo-2.7Bnumber of parameters=2.7B, zero-shot=true, scoring=logit scoring2022.12 | 61.1 | 54.6 | 50 | — | 67 | 52.3 | 50 | 37.5 | 59.1 | — | 60 | — | — | — | — | — | — | — | |
| Hybrid H3-2.7Bnumber of parameters=2.7B, zero-shot=true, scoring=logit scoring2022.12 | 61.1 | 56.8 | 37.5 | — | 81 | 57 | 51.3 | 36.5 | 58.7 | — | 71.3 | — | — | — | — | — | — | — | |
| GPT-2 medium (355M)number of parameters=355M, zero-shot=true, scoring=logit scoring2022.12 | 61 | 52.6 | 28.6 | — | 65 | 51.3 | 52 | 50 | 59.5 | — | 53.3 | — | — | — | — | — | — | — | |
| OPT-350Mnumber of parameters=350M, zero-shot=true, scoring=logit scoring2022.12 | 60.9 | 53.1 | 35.7 | — | 60 | 53.4 | 50.8 | 53.5 | 58.9 | — | 51.4 | — | — | — | — | — | — | — | |
| OPT-2.7Bnumber of parameters=2.7B, zero-shot=true, scoring=logit scoring2022.12 | 60.9 | 55.5 | 41.1 | — | 66 | 50.5 | 50.8 | 51 | 57.4 | — | 65.9 | — | — | — | — | — | — | — | |
| T0-3B# train tasks=35, Model Size=3B, Evaluation Protocol=In-Context Learning2024.03 | 60.1 | — | 48 | — | 56.8 | 51.3 | 54 | 53.5 | 56.2 | — | 20.5 | — | — | — | — | 65 | 36.1 | — | |
| GPT-Neo-1.3Bnumber of parameters=1.3B, zero-shot=true, scoring=logit scoring2022.12 | 59.9 | 52.1 | 33.9 | — | 66 | 52.3 | 50 | 41.3 | 57.9 | — | 55.5 | — | — | — | — | — | — | — | |
| OPT-125Mnumber of parameters=125M, zero-shot=true, scoring=logit scoring2022.12 | 59.6 | 50.1 | 37.4 | — | 60 | 48.7 | 52 | 39.4 | 58.9 | — | 44.9 | — | — | — | — | — | — | — | |
| Hybrid H3-125Mnumber of parameters=125M, zero-shot=true, scoring=logit scoring2022.12 | 59.6 | 53.9 | 48.2 | — | 67 | 59.2 | 51.4 | 39.4 | 51.4 | — | 55 | — | — | — | — | — | — | — | |
| GPT-Neo-125MShot=0-shot, Model Size=125M, Evaluation=rank classification2022.12 | 56.4 | 33.6 | 8.9 | — | 60 | 45.8 | 11.3 | 27.9 | 19.1 | — | 39.6 | — | — | — | — | — | — | — | |
| Hybrid H3-125Mshots=3-shot, evaluation_method=rank classification, parameters=125M2022.12 | 56.1 | 53.7 | 51.8 | — | 67 | 58.1 | 49.1 | 43.3 | 48.9 | — | 55 | — | — | — | — | — | — | — | |
| OPT-1.3Bnumber of parameters=1.3B, zero-shot=true, scoring=logit scoring2022.12 | 55 | 52.9 | 39.3 | — | 69 | 53.4 | 49.5 | 36.5 | 58.3 | — | 61.8 | — | — | — | — | — | — | — | |
| Hybrid H3-125MParameters=125M, Shots=3-shot, Mode=generation2022.12 | 49.5 | 50.5 | 41.1 | — | 67 | 58.1 | 49.1 | 43.3 | 40.3 | — | 55.2 | — | — | — | — | — | — | — | |
| Hybrid H3-125MShot=0-shot, Model Size=125M, Evaluation=rank classification2022.12 | 47.6 | 28.8 | 8.9 | — | 67 | 47.3 | 0 | 0 | 4.4 | — | 55 | — | — | — | — | — | — | — | |
| OPT-125MShot=0-shot, Model Size=125M, Evaluation=rank classification2022.12 | 45.9 | 41.7 | 8.9 | — | 60 | 49.8 | 48.4 | 36.5 | 39.1 | — | 44.9 | — | — | — | — | — | — | — | |
| Hybrid H3-355MParameters=355M, Shots=3-shot, Mode=generation2022.12 | 45.4 | 51.9 | 37.5 | — | 73 | 50.5 | 47.5 | 42.3 | 57.5 | — | 61.4 | — | — | — | — | — | — | — | |
| Hybrid H3-355MShot=0-shot, Model Size=355M, Evaluation=rank classification2022.12 | 44.4 | 36.6 | 8.9 | — | 69 | 47.3 | 21.5 | 22.1 | 17.1 | — | 62.3 | — | — | — | — | — | — | — | |
| OPT-125Mshots=3-shot, evaluation_method=rank classification, parameters=125M2022.12 | 41.9 | 47.9 | 44.6 | — | 60 | 47.3 | 50.2 | 36.5 | 57.9 | — | 44.9 | — | — | — | — | — | — | — | |
| OPT-350MShot=0-shot, Model Size=350M, Evaluation=rank classification2022.12 | 39.7 | 39.6 | 16.1 | — | 60 | 49.5 | 34.8 | 41.3 | 23.6 | — | 51.4 | — | — | — | — | — | — | — | |
| OPT-350MParameters=350M, Shots=3-shot, Mode=generation2022.12 | 39.4 | 45.6 | 41.1 | — | 59 | 46.2 | 50 | 37.5 | 40.6 | — | 51.3 | — | — | — | — | — | — | — | |
| OPT-125MParameters=125M, Shots=3-shot, Mode=generation2022.12 | 38.5 | 43.2 | 33.9 | — | 60 | 47.3 | 49.1 | 36.5 | 35.5 | — | 44.8 | — | — | — | — | — | — | — | |
| GPT-2 mediumShot=0-shot, Model Size=355M, Evaluation=rank classification2022.12 | 38.4 | 38.1 | 21.4 | — | 65 | 16.2 | 50.2 | 50 | 10.5 | — | 53.3 | — | — | — | — | — | — | — | |
| H3-125Mshots=3-shot, evaluation_method=rank classification, parameters=125M2022.12 | 37.8 | 43.9 | 48.2 | — | 51 | 52.3 | 50 | 63.5 | 32.6 | — | 15.8 | — | — | — | — | — | — | — | |
| H3-125MParameters=125M, Shots=3-shot, Mode=generation2022.12 | 37.8 | 20.3 | 8.9 | — | 53 | 47.3 | 0 | 0 | 0 | — | 15.4 | — | — | — | — | — | — | — | |
| GPT-2 medium (355M)Parameters=355M, Shots=3-shot, Mode=generation2022.12 | 37.8 | 44 | 28.6 | — | 63 | 47.3 | 50.5 | 36.5 | 35.3 | — | 53.1 | — | — | — | — | — | — | — | |
| DoRABackbone=Qwen0.5B, Type=Test Accuracy2026.05 | — | 68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Instruction-Queryable LoRABackbone=Qwen0.5B, Type=Test Accuracy2026.05 | — | 79.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=Qwen0.5B, Type=Test Accuracy2026.05 | — | 79.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Queryable LoRABackbone=Qwen0.5B, Type=Test Accuracy2026.05 | — | 79.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RepLoRABackbone=Qwen0.5B, Type=Test Accuracy2026.05 | — | 78.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |