Zero-shot Language Understanding Suite (LMB, HellA, PIQA, ARC-E/C, WINO, Open, MMLU)
83.4ARC-E AccuracyBLT Distill (Teacher)
Evaluation Results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BLT Distill (Teacher)Model=Llama 2 7B, Role=Teacher2026.02 | 83.4 | — | 80.7 | 80.7 | 55.2 | — | — | 66.3 | — | — | — | — | — | |
| Ours (Qwen 3 4B Teacher)Model=Qwen 3 4B, Role=Teacher (Baseline)2026.02 | 78.9 | 69.1 | 73.6 | 77.7 | 51.3 | 70.4 | 40.6 | 73 | — | — | — | — | — | |
| Ours (Stage 2)Model=Qwen 3 4B, Role=Student, Bytes=95B2026.02 | 77.1 | 66.2 | 69.7 | 75.4 | 52.6 | 66.4 | 41.2 | 68.5 | 2.19 | — | — | — | — | |
| ALM (Teacher)Model=Llama 3.2 3B, Role=Teacher2026.02 | 76.9 | — | — | — | — | 43.9 | — | 62.4 | — | — | — | — | — | |
| Ours (Stage 1)Model=Qwen 3 4B, Role=Student, Bytes=30B2026.02 | 76.8 | 68.4 | 72.5 | 77 | 50.2 | 69 | 40.2 | 66.8 | 1.71 | — | — | — | — | |
| Ours (Llama 3.2 3B Teacher)Model=Llama 3.2 3B, Role=Teacher (Baseline)2026.02 | 74.6 | 70.1 | 73.7 | 76.8 | 46 | 66.5 | 41.4 | 56 | — | — | — | — | — | |
| H-Net Distill (Teacher)Model=Llama 3.2 3B, Role=Teacher2026.02 | 74.5 | 70.1 | 73.7 | 76.8 | 46 | 66.5 | 41.4 | 56.1 | — | — | — | — | — | |
| ALM (Student)Model=Llama 3.2 3B, Role=Student (Byte)2026.02 | 73.7 | — | — | — | — | 40.1 | — | 55.9 | 4.5 | — | — | — | — | |
| Ours (Stage 1)Model=Llama 3.2 3B, Role=Student, Bytes=30B2026.02 | 73.6 | 69.5 | 72.9 | 76.1 | 43.9 | 67.2 | 41.1 | 51.6 | 1.15 | — | — | — | — | |
| H-Net Distill (Student)Model=Llama 3.2 3B, Role=Student (Byte), Bytes=189B2026.02 | 72.1 | 63.4 | 70.2 | 76.1 | 43.3 | 66.5 | 41.4 | 51.9 | 9.85 | — | — | — | — | |
| StableQATBits=4, Setting=20B Tokens & lr=1e-42026.01 | 72.05 | — | 71.28 | 77.58 | 44.97 | 67.64 | 42 | — | — | 70.21 | 91.5 | 67.15 | 2.67 | |
| ParetoQBits=4, Setting=20B Tokens & lr=1e-42026.01 | 71.83 | — | 71.2 | 76.58 | 45.48 | 66.25 | 42.4 | — | — | 70.13 | 90.6 | 66.81 | — | |
| BaselineBits=16, Setting=Baseline2026.01 | 71.63 | — | 73.61 | 77.48 | 45.99 | 69.85 | 43 | — | — | 73.39 | 92.7 | 68.46 | — | |
| ParetoQBits=3, Setting=20B Tokens & lr=1e-42026.01 | 71.47 | — | 70 | 76.15 | 45.16 | 65.93 | 42 | — | — | 70.28 | 91 | 66.5 | — | |
| StableQATBits=3, Setting=20B Tokens & lr=1e-42026.01 | 71.04 | — | 70.81 | 78.29 | 44.97 | 66.69 | 44 | — | — | 69.02 | 90.4 | 66.9 | 2.38 | |
| DSQBits=4, Setting=20B Tokens & lr=1e-42026.01 | 70.19 | — | 64.58 | 76.51 | 41.73 | 64.4 | 39.4 | — | — | 68.39 | 90.93 | 64.48 | — | |
| Ours (Stage 2)Model=Llama 3.2 3B, Role=Student, Bytes=95B2026.02 | 69.5 | 69.3 | 71.2 | 73.7 | 41.9 | 65.9 | 39.4 | 51.8 | 2.8 | — | — | — | — | |
| DSQBits=2, Setting=30B Tokens & lr=1e-42026.01 | 69.19 | — | 66.03 | 74.92 | 40.78 | 63.61 | 41.6 | — | — | 65.54 | 88.6 | 63.78 | — | |
| StableQATBits=2, Setting=30B Tokens & lr=1e-42026.01 | 68.48 | — | 65.2 | 75.41 | 40.87 | 63.38 | 41 | — | — | 63.06 | 87 | 63.05 | 1.36 | |
| BLT Distill (Student)Model=Llama 2 7B, Role=Student (Byte), Bytes=220B2026.02 | 66.6 | — | 76.1 | 77.4 | 45.8 | — | — | 63.7 | 7.34 | — | — | — | — | |
| DSQBits=3, Setting=20B Tokens & lr=1e-42026.01 | 66.58 | — | 66.58 | 76.28 | 41.64 | 64.48 | 40.6 | — | — | 70.24 | 89.8 | 64.52 | — | |
| ParetoQBits=2, Setting=30B Tokens & lr=1e-42026.01 | 65.73 | — | 64.13 | 74.68 | 38.08 | 61.43 | 39.4 | — | — | 65.73 | 84.3 | 61.69 | — | |
| Baseline no QATBits=4, Setting=Baseline no QAT2026.01 | 61.99 | — | 66.87 | 74.48 | 37.63 | 61.96 | 36.4 | — | — | 68.44 | 89.4 | 62.15 | — | |
| Baseline no QATBits=3, Setting=Baseline no QAT2026.01 | 26.14 | — | 26.61 | 52.5 | 25.43 | 49.01 | 28.4 | — | — | 44.01 | 26 | 34.76 | — | |
| Baseline no QATBits=2, Setting=Baseline no QAT2026.01 | 25.55 | — | 26.03 | 53.26 | 24.91 | 48.3 | 28.8 | — | — | 43.88 | 21.4 | 34.02 | — |