Zero-shot Reasoning and Language Modeling on Qwen1.5-MoE-A2.7B Evaluation Suite
6.79PPLbf16 baseline
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| bf16 baselineBit=bf162026.06 | 6.79 | 43.86 | 68.98 | 77.17 | 71.57 | 64.56 | 80.36 | 69.22 | 67.96 | |
| SliM-LLMBit=w4a162026.06 | 6.88 | 44.62 | 68.94 | 76.55 | 71.18 | 65.07 | 80.47 | 69.14 | 68 | |
| MoE-Salient-AQBit=w4a162026.06 | 6.89 | 44.37 | 69.91 | 76.92 | 71.38 | 64.27 | 80.2 | 68.82 | 67.98 | |
| MoE-Salient-AQBit=w3.5a162026.06 | 6.97 | 45.31 | 70 | 76.75 | 70.64 | 63.71 | 80.36 | 68.9 | 67.95 | |
| MxMoEBit=w3.5a162026.06 | 7.09 | 44.03 | 68.01 | 76.29 | 69.18 | 62.95 | 79.49 | 68.11 | 66.87 | |
| MxMoEBit=w4a162026.06 | 7.1 | 40.87 | 68.06 | 76.21 | 69.69 | 62.33 | 80.69 | 68.9 | 66.68 | |
| MoE-Salient-AQBit=w3a162026.06 | 7.16 | 43.69 | 67.55 | 75.94 | 69.36 | 63.15 | 79.22 | 66.61 | 66.5 | |
| SliM-LLMBit=w3a162026.06 | 7.27 | 40.36 | 64.86 | 74.94 | 69.47 | 63.79 | 78.94 | 65.75 | 65.44 | |
| MxMoEBit=w3a162026.06 | 7.43 | 43.43 | 67.21 | 75.17 | 68.41 | 60.76 | 79.49 | 67.64 | 66.02 | |
| MoE-Salient-AQBit=w2.5a162026.06 | 7.81 | 40.1 | 62.46 | 73.97 | 67.03 | 60.28 | 79.49 | 63.85 | 63.88 | |
| MxMoEBit=w2.5a162026.06 | 8.75 | 37.37 | 59.85 | 70.57 | 60.86 | 55.06 | 77.91 | 62.12 | 60.53 |