Language Understanding and Question Answering on SuperGLUE & SQuAD (val)
96SST-2 AccuracyAdam
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| AdamBackbone=Llama-3-8B2026.07 | 96 | 92 | 92 | 86.6 | 72.6 | 90.4 | |
| AdamBackbone=OPT-13B2026.07 | 95.3 | 80.9 | 94.6 | 83.5 | 66.3 | 89.5 | |
| LoRABackbone=Llama-3-8B2026.07 | 95 | 80.9 | 73.2 | 86.4 | 70.7 | 89.4 | |
| LoRABackbone=OPT-13B2026.07 | 94.8 | 78.3 | 69.6 | 80.2 | 64.3 | 88 | |
| ZO-ActBackbone=Llama-3-8B, Precision=Full, Optimizer=Adam2026.07 | 94.6 | 87 | 89.3 | 85.1 | 69.1 | 89.1 | |
| ZO-MuonBackbone=Llama-3-8B2026.07 | 94.3 | 81.2 | 69.6 | 82.9 | 65.2 | 88.2 | |
| ZO-ActBackbone=Llama-3-8B, Precision=Full, Optimizer=SGD2026.07 | 94 | 83.8 | 91.1 | 83.8 | 64.7 | 90.7 | |
| ZO-ActBackbone=OPT-13B, Precision=Full, Optimizer=Adam2026.07 | 94 | 74 | 75 | 73.1 | 59.4 | 85.1 | |
| HiZOOBackbone=Llama-3-8B2026.07 | 93.5 | 75.1 | 69.6 | 80 | 59.7 | 87.3 | |
| ZO-ActBackbone=Llama-3-8B, Precision=INT4, Optimizer=Adam2026.07 | 93.5 | 83.8 | 82.1 | 83.8 | 60.7 | 88.2 | |
| AGZOBackbone=Llama-3-8B2026.07 | 93.4 | 82.7 | 71.4 | 84.5 | 62.4 | 90.3 | |
| MeZOBackbone=Llama-3-8B2026.07 | 92.7 | 74.4 | 69.6 | 76.7 | 57.8 | 86.7 | |
| LOZOBackbone=Llama-3-8B2026.07 | 92.5 | 66.8 | 69.6 | 79.4 | 55.8 | 89 | |
| ZO-MuonBackbone=OPT-13B2026.07 | 92.5 | 72.9 | 71.4 | 72.4 | 61.7 | 84.5 | |
| ZO-ActBackbone=OPT-13B, Precision=Full, Optimizer=SGD2026.07 | 92.4 | 64.6 | 71.4 | 70 | 58.2 | 83.9 | |
| Subspace-MeZOBackbone=Llama-3-8B2026.07 | 92.3 | 68.6 | 69.6 | 80 | 62.9 | 84.5 | |
| S-MeZOBackbone=Llama-3-8B2026.07 | 92.1 | 69.7 | 69.6 | 80.5 | 56.9 | 87.5 | |
| SubZeroBackbone=Llama-3-8B2026.07 | 92.1 | 71.4 | 67.9 | 82 | 58.8 | 88.3 | |
| HiZOOBackbone=OPT-13B2026.07 | 92.1 | 69.3 | 69.6 | 67.6 | 59.4 | 82.1 | |
| SubZeroBackbone=OPT-13B2026.07 | 92.1 | 71.8 | 71.4 | 70.8 | 60.8 | 84.5 | |
| LOZOBackbone=OPT-13B2026.07 | 91.7 | 70.4 | 69.6 | 71.9 | 60.2 | 84.9 | |
| Subspace-MeZOBackbone=OPT-13B2026.07 | 91.7 | 70.7 | 71.4 | 68.1 | 61.7 | 83.5 | |
| MeZOBackbone=OPT-13B2026.07 | 91.4 | 66.1 | 66 | 66.1 | 59.4 | 81.8 | |
| S-MeZOBackbone=OPT-13B2026.07 | 90.4 | 63.5 | 69.6 | 66.4 | 58.8 | 80.8 | |
| AGZOBackbone=OPT-13B2026.07 | 89.8 | 67.5 | 66.1 | 68.1 | 56 | 85.4 |