Multiple Choice Question Answering on WinoG
68.9AccuracyMxMoE
Evaluation Results
| Method | Links | |
|---|---|---|
| MxMoEBackbone=Mixtral, Weight Bitwidth=2.252026.04 | 68.9 | |
| BaselineBackbone=Qwen1.5-MoE, Weight Bitwidth=162026.04 | 68.82 | |
| BaselineBackbone=Mixtral, Weight Bitwidth=162026.04 | 68.75 | |
| MoBiEBackbone=Mixtral, Weight Bitwidth=1.602026.04 | 68.09 | |
| GPTQBackbone=Mixtral, Weight Bitwidth=32026.04 | 66.28 | |
| NoWagBackbone=Mixtral, Weight Bitwidth=2.082026.04 | 65.51 | |
| MxMoEBackbone=Qwen1.5-MoE, Weight Bitwidth=2.252026.04 | 61.25 | |
| MTP (d=1)Horizon (d)=1, Parameters=1.3B, Training tokens=100B2025.11 | 60.54 | |
| GPTParameters=1.3B, Training tokens=100B2025.11 | 60.46 | |
| R2LMShots=5-sh, Candidate Type=long-target2026.06 | 60.2 | |
| GPTQBackbone=Qwen1.5-MoE, Weight Bitwidth=32026.04 | 59.99 | |
| JTP (d=2)Horizon (d)=2, Parameters=1.3B, Training tokens=100B2025.11 | 59.98 | |
| MoBiEBackbone=Qwen1.5-MoE, Weight Bitwidth=1.352026.04 | 59.75 | |
| R2LM-PI (plug-in)Shots=5-sh, Candidate Type=long-target2026.06 | 59.4 | |
| MTP (d=2)Horizon (d)=2, Parameters=1.3B, Training tokens=100B2025.11 | 59.35 | |
| NextLat (d=1)Horizon (d)=1, Parameters=1.3B, Training tokens=100B2025.11 | 59.27 | |
| NextLat (d=2)Horizon (d)=2, Parameters=1.3B, Training tokens=100B2025.11 | 59.2 | |
| JTP (d=1)Horizon (d)=1, Parameters=1.3B, Training tokens=100B2025.11 | 58.64 | |
| MoEQuantBackbone=Qwen1.5-MoE, Weight Bitwidth=22026.04 | 58.21 | |
| NoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.11 | 58.2 | |
| QuantMoe-BenchBackbone=Mixtral, Weight Bitwidth=2.32026.04 | 57.85 | |
| Bidirectional dLLMShots=5-sh, Candidate Type=long-target2026.06 | 57.7 | |
| RoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.11 | 57.1 | |
| Selective RoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.11 | 56 | |
| NoWagBackbone=Qwen1.5-MoE, Weight Bitwidth=2.082026.04 | 55.09 | |
| AWQBackbone=Mixtral, Weight Bitwidth=32026.04 | 54.98 | |
| Causal dLLMShots=5-sh, Candidate Type=long-target2026.06 | 54.3 | |
| Selective RoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B, Variant=stable_selective_rope2025.11 | 54.1 | |
| Selective RoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.11 | 53 | |
| NoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.11 | 52.8 | |
| RoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B2025.11 | 52.5 | |
| QuantMoe-BenchBackbone=Qwen1.5-MoE, Weight Bitwidth=2.32026.04 | 52.4 | |
| NoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.11 | 52.2 | |
| NoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B2025.11 | 52.2 | |
| MDLM2025.10 | 51.85 | |
| LDDM-M2025.10 | 51.46 | |
| Selective RoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.11 | 51.3 | |
| AWQBackbone=Qwen1.5-MoE, Weight Bitwidth=22026.04 | 50.91 | |
| RoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.11 | 50.8 | |
| RoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.11 | 50.7 | |
| GPTQBackbone=Qwen1.5-MoE, Weight Bitwidth=22026.04 | 50.43 | |
| GPTQBackbone=Mixtral, Weight Bitwidth=22026.04 | 50.33 | |
| AWQBackbone=Qwen1.5-MoE, Weight Bitwidth=32026.04 | 50.12 | |
| AWQBackbone=Mixtral, Weight Bitwidth=22026.04 | 50.08 | |
| Code specialistBackbone=Pythia-1B, Checkpoint=step10000, Freeze Layers=4, Seed=42, Samples=5002026.03 | 50 | |
| MoEQuantBackbone=Mixtral, Weight Bitwidth=22026.04 | 49.91 | |
| Base modelBackbone=Pythia-1B, Checkpoint=step10000, Freeze Layers=4, Seed=42, Samples=5002026.03 | 49.6 | |
| MonolithicBackbone=Pythia-1B, Checkpoint=step10000, Freeze Layers=4, Seed=42, Samples=5002026.03 | 49.4 | |
| KALAVAI MoEBackbone=Pythia-1B, Checkpoint=step10000, Freeze Layers=4, Seed=42, Samples=5002026.03 | 49.4 | |
| Weight averageBackbone=Pythia-1B, Checkpoint=step10000, Freeze Layers=4, Seed=42, Samples=5002026.03 | 48.6 | |
| Sci. specialistBackbone=Pythia-1B, Checkpoint=step10000, Freeze Layers=4, Seed=42, Samples=5002026.03 | 48.2 | |
| Fict. specialistBackbone=Pythia-1B, Checkpoint=step10000, Freeze Layers=4, Seed=42, Samples=5002026.03 | 48.2 |