Word Prediction on LAMBADA OpenAI
71.4AccuracyBaseline
Evaluation results
| Method | Links | |
|---|---|---|
| BaselineBackbone=Mixtral, Weight Bitwidth=162026.04 | 71.4 | |
| MoBiEBackbone=Mixtral, Weight Bitwidth=1.602026.04 | 71.05 | |
| BaselineBackbone=Qwen1.5-MoE, Weight Bitwidth=162026.04 | 70.83 | |
| MxMoEBackbone=Mixtral, Weight Bitwidth=2.252026.04 | 68.68 | |
| GPTQBackbone=Mixtral, Weight Bitwidth=32026.04 | 62.39 | |
| MxMoEBackbone=Qwen1.5-MoE, Weight Bitwidth=2.252026.04 | 56.43 | |
| NoWagBackbone=Mixtral, Weight Bitwidth=2.082026.04 | 50.46 | |
| MoBiEBackbone=Qwen1.5-MoE, Weight Bitwidth=1.352026.04 | 50.18 | |
| QwenModel Scale=Large (7B-1.5B), Base Model=Qwen, Method Configuration=Baseline2026.02 | 49.31 | |
| SD-MoEModel Scale=Large (7B-1.5B), Base Model=Qwen, Method Configuration=SD-MoE2026.02 | 47.29 | |
| GPTQBackbone=Qwen1.5-MoE, Weight Bitwidth=32026.04 | 46.18 | |
| SD-MoEModel Scale=Small (2B-0.8B), Base Model=Qwen, Method Configuration=SD-MoE2026.02 | 42.97 | |
| SD-MoEModel Scale=Small (2B-0.8B), Base Model=DeepSeek, Method Configuration=SD-MoE2026.02 | 40.21 | |
| DeepSeekModel Scale=Small (2B-0.8B), Base Model=DeepSeek, Method Configuration=Baseline2026.02 | 39.63 | |
| QwenModel Scale=Small (2B-0.8B), Base Model=Qwen, Method Configuration=Baseline2026.02 | 35.09 | |
| QuantMoe-BenchBackbone=Mixtral, Weight Bitwidth=2.32026.04 | 34.74 | |
| NoWagBackbone=Qwen1.5-MoE, Weight Bitwidth=2.082026.04 | 28.9 | |
| MoEQuantBackbone=Mixtral, Weight Bitwidth=22026.04 | 22.98 | |
| QuantMoe-BenchBackbone=Qwen1.5-MoE, Weight Bitwidth=2.32026.04 | 20.12 | |
| TransformerTraining=compute-matched, Training dataset=FineWeb-edu2026.05 | 15.13 | |
| MoEQuantBackbone=Qwen1.5-MoE, Weight Bitwidth=22026.04 | 12.03 | |
| AWQBackbone=Mixtral, Weight Bitwidth=32026.04 | 5.21 | |
| AWQBackbone=Qwen1.5-MoE, Weight Bitwidth=32026.04 | 3.29 | |
| SRMTraining=compute-matched, Training dataset=FineWeb-edu2026.05 | 3.05 | |
| MambaTraining=compute-matched, Training dataset=FineWeb-edu2026.05 | 1.01 | |
| AWQBackbone=Qwen1.5-MoE, Weight Bitwidth=22026.04 | 0 | |
| GPTQBackbone=Qwen1.5-MoE, Weight Bitwidth=22026.04 | 0 | |
| AWQBackbone=Mixtral, Weight Bitwidth=22026.04 | 0 | |
| GPTQBackbone=Mixtral, Weight Bitwidth=22026.04 | 0 |