Natural Language Understanding on GLUE (Hardware Latency and Size Profile)
84.3GLUE Macro ScoreT3C
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| T3CBackbone=RoBERTa-Base, Budget=Max2026.01 | 84.3 | 3.95 | 4.92 | 47.3 | 60 | 78.4 | 99.5 | 130 | 0.2 | |
| T3CBackbone=RoBERTa-Base, Budget=Med2026.01 | 84.2 | 3.82 | 4.78 | 45.6 | 57.8 | 76.5 | 97.4 | 118 | 0.17 | |
| MP-QATBackbone=RoBERTa-Base2026.01 | 84.1 | 4.26 | 5.18 | 57 | 70.6 | 96.2 | 121.7 | 372 | 0.27 | |
| T3CBackbone=RoBERTa-Base, Budget=Tiny2026.01 | 83.9 | 3.68 | 4.64 | 43.9 | 55.7 | 73.8 | 94 | 108 | 0.19 | |
| T3CBackbone=BERT-Base, Budget=Max2026.01 | 82.4 | 3.26 | 4.16 | 44.3 | 56.6 | 74 | 94.2 | 112 | 0.21 | |
| T3CBackbone=BERT-Base, Budget=Med2026.01 | 82.2 | 3.18 | 4.06 | 43.1 | 55 | 72.4 | 92.3 | 101 | 0.18 | |
| QATBackbone=BERT-Base, Bit-width=8b2026.01 | 82 | 3.7 | 4.7 | 55.8 | 69.4 | 92 | 118 | 338 | 0.29 | |
| T3CBackbone=BERT-Base, Budget=Tiny2026.01 | 81.8 | 3.4 | 4.35 | 41 | 52.6 | 69.2 | 88.1 | 92 | 0.2 | |
| T3CBackbone=DistilBERT, Budget=Max2026.01 | 80 | 2.52 | 3.16 | 32.1 | 41 | 53.2 | 67.7 | 69 | 0.23 | |
| T3CBackbone=DistilBERT, Budget=Med2026.01 | 79.9 | 2.44 | 3.06 | 31 | 39.6 | 51.8 | 65.8 | 63 | 0.2 | |
| PTQBackbone=DistilBERT, Bit-width=4b2026.01 | 79.8 | 2.66 | 3.28 | 36.4 | 45.2 | 59.8 | 76 | 94 | 0.33 | |
| T3CBackbone=DistilBERT, Budget=Tiny2026.01 | 79.5 | 2.36 | 2.98 | 29.8 | 38 | 49.9 | 63.5 | 58 | 0.22 |