Scientific Reasoning on GPQA (Accuracy, Mean, Drop)
73.74AccuracySaw-INT4
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Saw-INT4Model=GLM-4.7 FP8, 358B, BPE=4.252026.05 | 73.74 | 77.95 | 0.06 | |
| OSCARModel=GLM-4.7 FP8, 358B, BPE=2.282026.05 | 73.57 | 78.16 | 0.27 | |
| BF16Model=GLM-4.7 FP8, 358B, BPE=16.002026.05 | 73.23 | 77.89 | — | |
| QuaRot-INT2Model=GLM-4.7 FP8, 358B, BPE=2.252026.05 | 68.01 | 75.14 | -2.75 | |
| BF16Model=Qwen3-4B Thinking-2507, BPE=16.002026.05 | 67.27 | 75.64 | — | |
| TurboQuant (no MP)Model=GLM-4.7 FP8, 358B, BPE=3.252026.05 | 66.67 | 78.15 | 0.26 | |
| Saw-INT4Model=Qwen3-4B Thinking-2507, BPE=4.252026.05 | 66.37 | 73.11 | -2.53 | |
| OSCARModel=Qwen3-4B Thinking-2507, BPE=2.282026.05 | 64.95 | 71.864 | -3.78 | |
| OSCARModel=Qwen3-32B, BPE=2.282026.05 | 60.4 | 74.17 | -0.02 | |
| Saw-INT4Model=Qwen3-32B, BPE=4.252026.05 | 59.29 | 74.43 | 0.24 | |
| TurboQuant (no MP)Model=Qwen3-32B, BPE=3.252026.05 | 58.69 | 71.99 | -2.2 | |
| BF16Model=Qwen3-32B, BPE=16.002026.05 | 58.49 | 74.19 | — | |
| BF16Model=Qwen3-8B, BPE=16.002026.05 | 56.67 | 70.84 | — | |
| TurboQuant (no MP)Model=Qwen3-8B, BPE=3.252026.05 | 55.05 | 56.88 | -13.96 | |
| OSCARModel=Qwen3-8B, BPE=2.282026.05 | 55.05 | 69.416 | -1.42 | |
| Saw-INT4Model=Qwen3-8B, BPE=4.252026.05 | 54.85 | 69.97 | -0.87 | |
| Naive INT2Model=GLM-4.7 FP8, 358B, BPE=2.252026.05 | 54.55 | 60.49 | -17.4 | |
| TurboQuant (no MP)Model=Qwen3-4B Thinking-2507, BPE=3.252026.05 | 41.41 | 31.74 | -43.9 | |
| QuaRot-INT2Model=Qwen3-32B, BPE=2.252026.05 | 19.7 | 7.9 | -66.29 | |
| QuaRot-INT2Model=Qwen3-8B, BPE=2.252026.05 | 14.98 | 10.14 | -60.7 | |
| QuaRot-INT2Model=Qwen3-4B Thinking-2507, BPE=2.252026.05 | 0.34 | 1.4 | -74.24 | |
| Naive INT2Model=Qwen3-4B Thinking-2507, BPE=2.252026.05 | 0 | 0 | -75.64 | |
| Naive INT2Model=Qwen3-8B, BPE=2.252026.05 | 0 | 0 | -70.84 | |
| Naive INT2Model=Qwen3-32B, BPE=2.252026.05 | 0 | 0 | -74.19 |