Common-sense Reasoning Suite (BoolQ, SciQ, PIQA, WinoG., ARC-C, HellaS.)
88BoolQ AccuracyDense
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| DenseBase Model=Qwen2.5 7B, Active Params=7.62B, #FT Tokens=18T*2026.06 | 88 | 96.6 | 78.6 | 76.2 | 64.1 | 80.3 | 80.6 | |
| DenseBase Model=LLaMA-3 8B, Active Params=8.03B, #FT Tokens=15T*2026.06 | 83.2 | 96.2 | 79.6 | 77.3 | 58.3 | 82.1 | 79.4 | |
| DenseBase Model=LLaMA-2 7B, Active Params=6.74B, #FT Tokens=2T*2026.06 | 82 | 94 | 78.1 | 74.3 | 52.5 | 78.9 | 76.6 | |
| DOT-MoEBase Model=Qwen2.5 7B, Active Params=4.76B, #FT Tokens=1.2B2026.06 | 81.3 | 94.6 | 73.1 | 68.1 | 49.8 | 67.3 | 73.4 | |
| LLaMA-MoE-v2 (E8A1S1)Base Model=LLaMA-3 8B, Active Params=3.80B, #FT Tokens=7B2026.06 | 76.9 | 92.8 | 67.9 | 58.4 | 40.2 | 53.7 | 65 | |
| DOT-MoEBase Model=LLaMA-3 8B, Active Params=3.80B, #FT Tokens=7B2026.06 | 75.4 | 96.2 | 73.3 | 66.1 | 49.1 | 66 | 71 | |
| DOT-MoEBase Model=Qwen2.5 7B, Active Params=3.34B, #FT Tokens=1.2B2026.06 | 75.2 | 95 | 70.4 | 61.3 | 42.8 | 58.4 | 67.2 | |
| DOT-MoEBase Model=LLaMA-3 8B, Active Params=3.80B, #FT Tokens=1.2B2026.06 | 75 | 94.2 | 70.2 | 63.8 | 42.4 | 61.1 | 67.8 | |
| LLaMA-MoE-v2 (E8A2)Base Model=LLaMA-3 8B, Active Params=3.80B, #FT Tokens=7B2026.06 | 74.6 | 94.5 | 69.3 | 60.5 | 42.8 | 59 | 66.8 | |
| DOT-MoEBase Model=LLaMA-2 7B, Active Params=3.49B, #FT Tokens=1.2B2026.06 | 72.5 | 94.3 | 69.3 | 62.5 | 40.9 | 60.2 | 66.6 | |
| CMoE (E8A1S1)Base Model=LLaMA-3 8B, Active Params=3.80B, #FT Tokens=1.2B2026.06 | 71.1 | 94.4 | 69.5 | 59.5 | 38.2 | 55.3 | 64.7 | |
| CMoE (E8A1S1)Base Model=Qwen2.5 7B, Active Params=3.34B, #FT Tokens=1.2B2026.06 | 69.4 | 93.8 | 67.6 | 59 | 37.8 | 57.2 | 64.1 | |
| CMoE (E8S1A1)Base Model=LLaMA-2 7B, Active Params=3.49B, #FT Tokens=1.2B2026.06 | 55 | 77.5 | 57.1 | 54.1 | 27.6 | 38.8 | 51.7 | |
| LLaMA-MoE-v2 (E8A2)Base Model=LLaMA-2 7B, Active Params=3.49B, #FT Tokens=1.2B2026.06 | 51.3 | 67 | 56.6 | 52.9 | 25.7 | 35.1 | 48.1 | |
| LLaMA-MoE (E8A2)Base Model=LLaMA-2 7B, Active Params=3.49B, #FT Tokens=1.2B2026.06 | 37.8 | 20 | 49.7 | 50.1 | 25.8 | 26.2 | 34.9 |