Natural Language Understanding on GLUE v1 (dev)
93.8MRPC ScoreAutoBERT-Zero*
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AutoBERT-Zero*#Params=104M, Infer FLOPs=2.3e10, initialization=supernet2021.07 | 93.8 | 67.3 | 86.4 | 86.3 | 90.8 | 85.2 | 91.7 | 92.5 | 95.2 | 87.7 | — | |
| AutoBERT-Zero#Params=104M, Infer FLOPs=2.3e102021.07 | 93.3 | 64.5 | 85.5 | 85.3 | 90.8 | 81.9 | 88.9 | 92 | 94.2 | 86.3 | — | |
| AutoBERT-w/o-desc#Params=104M, Infer FLOPs=2.3e10, architecture=backbone without descending kernel sizes2021.07 | 92.8 | 65.1 | 84.5 | 85 | 90.5 | 78.7 | 88.2 | 91.6 | 93.7 | 85.6 | — | |
| AutoBERT-conv#Params=104M, Infer FLOPs=2.2e10, architecture=three continuous convolution layers2021.07 | 92.6 | 63.8 | 84.4 | 84.6 | 90.1 | 80.5 | 88.3 | 91.7 | 93.5 | 85.5 | — | |
| AutoBERT-att#Params=104M, Infer FLOPs=2.3e10, architecture=three continuous attention layers2021.07 | 92.2 | 65.4 | 84.6 | 85 | 90.4 | 81.6 | 88.5 | 91.8 | 93.8 | 85.9 | — | |
| BERTbase (Teacher)#Params=110M2023.05 | 91.4 | 60.3 | 84.9 | 84.8 | 89.4 | 69.7 | 91.5 | 91.7 | 93.7 | 84.1 | — | |
| AD-KD#Params=66M2023.05 | 91.2 | 58.3 | 83.4 | 84.2 | 89.2 | 70.9 | 91.2 | 91.2 | 91.9 | 83.4 | — | |
| MGSKD#Params=66M2023.05 | 89.8 | 49.1 | 83.3 | 83.9 | 88.5 | 67.9 | 91.2 | 90.3 | 91.7 | 81.5 | — | |
| BERT-base(ours)#Params=110M, Infer FLOPs=2.9e10, implementation=reproduced by current paper authors2021.07 | 89.7 | 58.1 | 84.8 | 85.2 | 88.8 | 69 | 88.2 | 91.5 | 92.9 | 83.1 | — | |
| Flexible Multi-Task Model (w/o KD)Layers=7 + 60, Overhead=67 (69.8%)2021.07 | 89.7 | 55.2 | — | — | 88.9 | 71.5 | 91.1 | 91.7 | 93.1 | 83.2 | 84.5 | |
| CKD#Params=66M2023.05 | 89.6 | 55.1 | 83.6 | 84.1 | 89 | 67.3 | 91.2 | 90.5 | 93 | 82.4 | — | |
| Vanilla KD#Params=66M2023.05 | 89.4 | 53.6 | 82.7 | 83.1 | 88.7 | 66.8 | 90.5 | 90.1 | 91.1 | 81.6 | — | |
| PD#Params=66M2023.05 | 89.4 | — | 82.5 | 83.4 | — | 66.7 | 90.7 | 89.4 | 91.1 | — | — | |
| BERT6 (Student)#Params=66M2023.05 | 89.2 | 51.2 | 81.7 | 82.6 | 88.3 | 66.1 | 90.4 | 89.3 | 91 | 80.9 | — | |
| BERT-of-TheseusLayers=6 x 8, Overhead=48 (50.0%)2021.07 | 89 | 51.1 | — | — | 88.7 | 68.2 | 89.6 | 89.5 | 91.5 | 81.2 | 82.3 | |
| TinyBERT#Params=66M2023.05 | 88.8 | 53.8 | 83.1 | 83.4 | 88.3 | 66.9 | 90.5 | 89.9 | 92.3 | 81.7 | — | |
| TinyBERTLayers=6 x 8, Overhead=48 (50.0%)2021.07 | 88.4 | 42.8 | — | — | — | 72.2 | 90.6 | 90.5 | 91.6 | — | 83.5 | |
| Full fine-tuningLayers=12 x 8, Overhead=96 (100%)2021.07 | 88.2 | 54.7 | — | — | 88.8 | 69.7 | 91.1 | 91.6 | 93.4 | 82.8 | 84.6 | |
| Flexible Multi-Task Model (KD-3)Layers=7 + 3 x 8, Overhead=31 (32.3%)2021.07 | 88 | 50 | — | — | 88.9 | 66.8 | 91.2 | 90.2 | 92.7 | 81.3 | 82.9 | |
| Flexible Multi-Task Model (mixed)Layers=7 + 26, Overhead=33 (34.3%)2021.07 | 88 | 55.2 | — | — | 88.3 | 71.5 | 91 | 90.2 | 92.7 | 82.5 | 82.9 | |
| DistillBERTLayers=6 x 8, Overhead=48 (50.0%)2021.07 | 87.5 | 51.3 | — | — | 86.9 | 59.9 | 88.5 | 89.2 | 91.3 | 79.6 | 82.2 | |
| MT-DNN (full)Layers=12 x 1, Overhead=12 (12.5%)2021.07 | 87.4 | 51.3 | — | — | 86.8 | 80.9 | 87.6 | 91.1 | 93.5 | 82.9 | 84.4 | |
| PD-BERTLayers=6 x 8, Overhead=48 (50.0%)2021.07 | 87.2 | — | — | — | — | 66.7 | 89.1 | 89 | 91.1 | — | 83 | |
| Flexible Multi-Task Model (KD-2)Layers=7 + 2 x 8, Overhead=23 (24.0%)2021.07 | 86.3 | 44 | — | — | 88.6 | 64.6 | 91.3 | 88.6 | 92.7 | 79.7 | 81.7 | |
| Vanilla-KDLayers=6 x 8, Overhead=48 (50.0%)2021.07 | 86.2 | 45.1 | — | — | 84.9 | 64.9 | 88.1 | 88 | 90.5 | 78.5 | 80.1 | |
| BERT-PKDLayers=6 x 8, Overhead=48 (50.0%)2021.07 | 85.7 | 45.5 | — | — | 86.2 | 66.5 | 88.4 | 88.4 | 91.3 | 79.2 | 81.3 | |
| MiniLMLayers=6 x 8, Overhead=48 (50.0%)2021.07 | 85.7 | 45.5 | — | — | 86.2 | 66.5 | 88.4 | 88.4 | 91.3 | 79.2 | 81.3 | |
| PKD#Params=66M2023.05 | 85.7 | 45.5 | 81.3 | — | 86.2 | 66.5 | 88.4 | 88.4 | 91.3 | 79.2 | — | |
| Flexible Multi-Task Model (KD-1)Layers=7 + 1 x 8, Overhead=15 (15.6%)2021.07 | 85.1 | 36.4 | — | — | 88.3 | 66.1 | 91 | 86.4 | 90.7 | 77.4 | 77.5 | |
| MT-DNN (LOO)2021.07 | 74.2 | 10.2 | — | — | 72.9 | 60.6 | 66.5 | 69.7 | 79.2 | — | 56.7 |