Word Prediction on LAMBADA standard
65.57AccuracyBaseline
Evaluation results
| Method | Links | |
|---|---|---|
| BaselineBackbone=Qwen1.5-MoE, Weight Bitwidth=162026.04 | 65.57 | |
| BaselineBackbone=Mixtral, Weight Bitwidth=162026.04 | 64.49 | |
| MxMoEBackbone=Mixtral, Weight Bitwidth=2.252026.04 | 62.18 | |
| MoBiEBackbone=Mixtral, Weight Bitwidth=1.602026.04 | 58.24 | |
| GPTQBackbone=Mixtral, Weight Bitwidth=32026.04 | 54.19 | |
| MxMoEBackbone=Qwen1.5-MoE, Weight Bitwidth=2.252026.04 | 51 | |
| MoBiEBackbone=Qwen1.5-MoE, Weight Bitwidth=1.352026.04 | 38.4 | |
| NoWagBackbone=Mixtral, Weight Bitwidth=2.082026.04 | 36.7 | |
| GPTQBackbone=Qwen1.5-MoE, Weight Bitwidth=32026.04 | 35.47 | |
| QuantMoe-BenchBackbone=Mixtral, Weight Bitwidth=2.32026.04 | 30.11 | |
| NoWagBackbone=Qwen1.5-MoE, Weight Bitwidth=2.082026.04 | 22.92 | |
| MoEQuantBackbone=Mixtral, Weight Bitwidth=22026.04 | 18.94 | |
| QuantMoe-BenchBackbone=Qwen1.5-MoE, Weight Bitwidth=2.32026.04 | 16.77 | |
| MoEQuantBackbone=Qwen1.5-MoE, Weight Bitwidth=22026.04 | 8.05 | |
| AWQBackbone=Mixtral, Weight Bitwidth=32026.04 | 3.93 | |
| AWQBackbone=Qwen1.5-MoE, Weight Bitwidth=32026.04 | 0.18 | |
| AWQBackbone=Qwen1.5-MoE, Weight Bitwidth=22026.04 | 0 | |
| GPTQBackbone=Qwen1.5-MoE, Weight Bitwidth=22026.04 | 0 | |
| AWQBackbone=Mixtral, Weight Bitwidth=22026.04 | 0 | |
| GPTQBackbone=Mixtral, Weight Bitwidth=22026.04 | 0 |