Natural Language Understanding on SuperGLUE (dev)
93.2Average ScoreST-MoE 32B
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ST-MoE 32BTotal Parameters=269B, Effective Parameters=32B, Evaluation Protocol=Finetuned2022.05 | 93.2 | 95.7 | 93.1 | — | 100 | 100 | 81 | 100 | 95 | — | 100 | 90.4 | 69.9 | 95.6 | — | |
| ST-MoE-32BModel Parameters=32B2022.04 | 93.2 | 95.7 | 93.1 | — | 100 | 100 | 81 | 100 | 95 | — | 100 | 90.4 | 69.9 | 95.6 | — | |
| PaLM 540BParameters=540B, Evaluation Protocol=Finetuned2022.05 | 92.6 | 95.7 | 92.2 | — | 100 | 100 | 78.8 | 100 | 94 | — | 100 | 90.1 | 69.2 | 94.6 | — | |
| PaLM 540BModel Parameters=540B, Evaluation Protocol=finetuned2022.04 | 92.6 | 95.7 | 92.2 | — | 100 | 100 | 78.8 | 100 | 94 | — | 100 | 90.1 | 69.2 | 94.6 | — | |
| UL2 20BParameters=20B, Evaluation Protocol=Finetuned2022.05 | 90.7 | 92.1 | 90.8 | — | 99 | 98.7 | 77.3 | 98.1 | 93.7 | — | 98.2 | 88.4 | 64.8 | 93.2 | — | |
| T5 11BParameters=11B, Evaluation Protocol=Finetuned2022.05 | 89.9 | 93.9 | 90.8 | — | 98 | 94.9 | 77.3 | 96.2 | 93.8 | — | 96.4 | 87.4 | 66.1 | 93.2 | — | |
| T5-11BModel Parameters=11B2022.04 | 89.9 | 93.9 | 90.8 | — | 98 | 94.9 | 77.3 | 96.2 | 93.8 | — | 96.4 | 87.4 | 66.1 | 93.2 | — | |
| PaLM 62BParameters=62B, Evaluation Protocol=Finetuned2022.05 | 89.2 | 89.5 | 90.6 | — | 98 | 96.4 | 75.9 | 96.2 | 93 | — | 95.7 | 87.7 | 61.9 | 92.4 | — | |
| GLMTeacher Scale=T32023.06 | 89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLMD-vcTeacher-Student Scale=T3-S3, Student parameters=2B, Excludes=ReCoRD2023.06 | 84.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PaLM 8BParameters=8B, Evaluation Protocol=Finetuned2022.05 | 83.4 | 84.5 | 87.6 | — | 86 | 96.4 | 73.4 | 88.5 | 89.7 | — | 92.1 | 81.6 | 64 | 89.3 | — | |
| GLM_RoBERTaPre-trained corpora=Larger corpora2021.03 | 82.9 | 87.7 | 84.7 | — | 82 | 98.2 | 71.2 | 83.7 | 89.6 | 89 | 98.7 | 82.4 | 50.1 | — | — | |
| RoBERTa_LargePre-trained corpora=Larger corpora2021.03 | 81.5 | 87 | 86.1 | — | 90 | 94.6 | 72.6 | 63.5 | 89 | 88.4 | 96.1 | 84.4 | 52.9 | — | — | |
| T5_LargePre-trained corpora=Larger corpora2021.03 | 81.2 | 84.8 | 84.3 | — | 78 | 98.2 | 71.6 | 84.6 | 85.7 | 85 | 96.4 | 80.9 | 46.6 | — | — | |
| GLM_515MPre-trained corpora=BookCorpus and Wikipedia, Backbone Scale=1.5x BERT_Large2021.03 | 78.8 | 79.1 | 81.3 | — | 85 | 96.4 | 69.4 | 81.7 | 82.3 | 81.7 | 95 | 77.2 | 35 | — | — | |
| PaLM 540BEvaluation Mode=Zero-shot2022.08 | 78.8 | 72.9 | 88 | — | 93 | 51.8 | 59.1 | 89.1 | — | 92.9 | — | 83.5 | — | — | — | |
| GLM_410MPre-trained corpora=BookCorpus and Wikipedia, Backbone Scale=1.25x BERT_Large2021.03 | 78 | 79.4 | 81.9 | — | 80 | 96.4 | 69 | 81.7 | 81.5 | 80.9 | 93.2 | 76.2 | 35.5 | — | — | |
| GLMTeacher Scale=T22023.06 | 77.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLM_LargePre-trained corpora=BookCorpus and Wikipedia2021.03 | 77 | 74 | 82.1 | — | 76 | 94.6 | 68.5 | 81.7 | 81.7 | 81.1 | 96.1 | 77.1 | 36.3 | — | — | |
| GLM_SentPre-trained corpora=BookCorpus and Wikipedia2021.03 | 76.8 | 79.1 | 80.8 | — | 77 | 93.7 | 70.4 | 79.8 | 80.7 | 80.2 | 94.6 | 76.9 | 36.1 | — | — | |
| T5_BasePre-trained corpora=Larger corpora2021.03 | 76 | 78.3 | 80.8 | — | 73 | 92.9 | 67.9 | 79.8 | 76.2 | 75.4 | 94.8 | 76.4 | 40 | — | — | |
| BART_LargePre-trained corpora=Larger corpora2021.03 | 76 | 84.5 | 84.3 | — | 60 | 92.9 | 69 | 65.4 | 88.3 | 87.8 | 90.5 | 81.8 | 48 | — | — | |
| GLM_DocPre-trained corpora=BookCorpus and Wikipedia2021.03 | 75.7 | 76.2 | 79.8 | — | 77 | 96.4 | 63.6 | 78.8 | 80.2 | 79.6 | 97.3 | 74.6 | 32.1 | — | — | |
| UniLM_LargePre-trained corpora=BookCorpus and Wikipedia2021.03 | 74.1 | 76.5 | 80.5 | — | 72 | 91.1 | 69.7 | 65.4 | 80 | 79.1 | 91 | 77.2 | 38.2 | — | — | |
| GLMDTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=55M2023.06 | 72.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLMD-vcTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 72.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERT_LargePre-trained corpora=BookCorpus and Wikipedia2021.03 | 72 | 73.6 | 80.1 | — | 69 | 92.9 | 71 | 64.4 | 76.3 | 75.6 | 94.8 | 71.9 | 24.1 | — | — | |
| GLMTeacher Scale=T12023.06 | 71.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLMD-vcTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 71.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLMDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=55M2023.06 | 70.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TinyBERTTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 70.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLM_BasePre-trained corpora=BookCorpus and Wikipedia2021.03 | 70.7 | 71.2 | 77 | — | 71 | 85.7 | 64.7 | 72.1 | 73.5 | 72.8 | 89.5 | 72.1 | 26.1 | — | — | |
| AlexaTM 20BEvaluation Mode=Zero-shot2022.08 | 69.16 | 68.59 | 69.44 | — | 78 | 67.9 | 53.29 | 68.27 | — | 88.4 | — | 59.57 | — | — | — | |
| TinyBERTTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 69.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLMD-vcTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 67.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLMDTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=16M2023.06 | 67.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLMD+alTeacher-Student Scale=T1-S1, rv=0.5, Student parameters=14M2023.06 | 67.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TinyBERTTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 67.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLMD-vc+alTeacher-Student Scale=T1-S1, rv=0.5, Student parameters=16M2023.06 | 67.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLMD-vc+moTeacher-Student Scale=T1-S1, rv=0.5, Student parameters=16M2023.06 | 67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DIITOTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 66.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PKDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 66.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Universal-KDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 66.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERT_BasePre-trained corpora=BookCorpus and Wikipedia2021.03 | 66.1 | 70 | 74.9 | — | 66 | 76.8 | 68.8 | 65.4 | 65.4 | 64.9 | 70.9 | 68.4 | 21.5 | — | — | |
| TheseusTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 66.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MiniLMv2Teacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 66.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TMKDTeacher-Student Scale=(T1, T2)-S2, Student parameters=66M2023.06 | 65.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UncertaintyTeacher-Student Scale=(T1, T2)-S2, Student parameters=66M2023.06 | 65.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RL-KDTeacher-Student Scale=(T1, T2)-S2, Student parameters=66M2023.06 | 65.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MobileBERT25MTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 65.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TinyBERTTeacher-Student Scale=T3-S3, Student parameters=2B, Excludes=ReCoRD2023.06 | 65.09 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ALP-KDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 64.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistilBERTTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 63.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MiniLMTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 63.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MiniLMv2Teacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 63.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RAIL-KDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 63.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MGSKDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 63.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Annealing-KDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 63.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CKDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 63.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Annealing-KDTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 63.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UL 20BEvaluation Mode=Zero-shot2022.08 | 63 | 60.7 | 63.1 | — | 85 | 41.1 | 49.8 | 79.9 | — | 88.1 | — | 36.2 | — | — | — | |
| Continuation-KDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 62.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CKDTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 62.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MiniLMTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 62.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LRC-BERTTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 62.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MiniLMv2Teacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 62.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CKDTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 61.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Continuation-KDTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 61.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MiniLMTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 61.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PDTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 61.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RAIL-KDTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 61.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT3 175BEvaluation Mode=Zero-shot2022.08 | 61.2 | 63.5 | 60.5 | — | 91 | 46.4 | 0 | 65.4 | — | 90.2 | — | 72.9 | — | — | — | |
| PDTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Annealing-KDTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Continuation-KDTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LRC-BERTTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 60.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RAIL-KDTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 60.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MGSKDTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 59.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MT-BERTTeacher-Student Scale=(T1, T2)-S2, Student parameters=66M2023.06 | 59.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MGSKDTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 58.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LRC-BERTTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 58.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT3 13BEvaluation Mode=Zero-shot2022.08 | 57.2 | 62.8 | 66.2 | — | 84 | 19.6 | 0 | 64.4 | — | 89 | — | 71.4 | — | — | — | |
| SIDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 55.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Universal-KDTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 54.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TAKDTeacher-Student Scale=T2-A1-A2-S2, Student parameters=66M2023.06 | 54.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DGKDTeacher-Student Scale=T2-A1-A2-S2, Student parameters=66M2023.06 | 54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KDTeacher-Student Scale=T2-S2, Compression ratio=0.2, Student parameters=66M2023.06 | 53.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Universal-KDTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 52.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KDTeacher-Student Scale=T1-S2, Compression ratio=0.5, Student parameters=66M2023.06 | 52.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KDTeacher-Student Scale=T1-S1, Compression ratio=0.2, Student parameters=22M2023.06 | 52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BasedArchitecture=Based, shots=1-shot2024.02 | — | 47.6 | 48.3 | — | 66 | — | 46.5 | 55.8 | 62.7 | — | — | — | — | 62.1 | 51.5 | |
| BaselineBackbone=BERT-large2023.05 | — | — | 78.1 | — | 72 | 91.1 | — | — | — | — | — | 70.3 | — | — | — | |
| BaselineBackbone=BERT-base2023.05 | — | — | 74.4 | — | 63 | 83.9 | — | — | — | — | — | 68.1 | — | — | — | |
| BLOOM 175BEvaluation Mode=Zero-shot2022.08 | — | 52 | 63.5 | — | 56 | 33.9 | 50.6 | 51.9 | — | — | — | 57.1 | — | — | — | |
| E-LANGArchitecture=T5, Super model=T5-11B, Swift model=T5-Large, Avg. Speed-up (FLOPs)=2.9X, Avg. Speed-up (time)=2.0X2022.03 | — | 93.2 | 88.7 | 84.9 | 90 | 93.1 | 78.1 | 89.4 | — | — | — | — | — | — | — | |
| ReBasedArchitecture=ReBased, shots=1-shot2024.02 | — | 53.1 | 52 | — | 67 | — | 46.9 | 56.7 | 62.8 | — | — | — | — | 62.2 | 51.9 | |
| SCTDBackbone=BERT-large2023.05 | — | — | 79.7 | — | 72 | 92.9 | — | — | — | — | — | 72.8 | — | — | — | |
| SCTDBackbone=BERT-base2023.05 | — | — | 73.8 | — | 68 | 87.5 | — | — | — | — | — | 68.9 | — | — | — | |
| Super (11B)Architecture=T5-11B, FLOPs=87e112022.03 | — | 93.1 | 89.4 | 84.9 | 93 | 93.1 | 77.4 | 89.4 | — | — | — | — | — | — | — |