Reading Comprehension on RACE (Middle and High Difficulty Scores)
70.2RACE Middle ScoreCAD
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| CADBase Model=LLaMA3-8B-Instruct2025.03 | 70.2 | 54.3 | — | |
| HICDBase Model=Mistral-7B-v0.32025.03 | 68.9 | 55.7 | — | |
| DoLABase Model=LLaMA3-8B-Instruct2025.03 | 68.5 | 52.7 | — | |
| Mistral-7B-v0.3Base Model=Mistral-7B-v0.32025.03 | 67.7 | 54.1 | — | |
| DoLABase Model=Mistral-7B-v0.32025.03 | 67.1 | 54.3 | — | |
| LLaMA3-8B-InstructBase Model=LLaMA3-8B-Instruct2025.03 | 67.1 | 51.8 | — | |
| HICDBase Model=LLaMA3-8B-Instruct2025.03 | 66.9 | 54.9 | — | |
| CADBase Model=Mistral-7B-v0.32025.03 | 66.7 | 56.1 | — | |
| HICDBase Model=Qwen-7B2025.03 | 57.3 | 47.1 | — | |
| CADBase Model=Qwen-7B2025.03 | 57.1 | 46.2 | — | |
| DoLABase Model=Qwen-7B2025.03 | 56.6 | 44.5 | — | |
| Qwen-7BBase Model=Qwen-7B2025.03 | 56.1 | 44.7 | — | |
| Expert Divergence LearningModel Size=15B-A1.5B, Scheme=49-class2026.02 | 34.54 | 28.76 | — | |
| MoEModel Size=15B-A1.5B, Training Strategy=Baseline2026.02 | 33.36 | 28.64 | — | |
| Expert Divergence LearningModel Size=15B-A1.5B, Scheme=3-class2026.02 | 32.94 | 28.1 | — | |
| Expert Divergence LearningModel Size=8B-A0.8B, Scheme=3-class2026.02 | 32.8 | 28.62 | — | |
| MoEModel Size=8B-A0.8B, Training Strategy=Baseline2026.02 | 32.52 | 28.42 | — | |
| Expert Divergence LearningModel Size=3B-A0.3B, Scheme=49-class2026.02 | 32.03 | 28.27 | — | |
| MoEModel Size=3B-A0.3B, Training Strategy=Baseline2026.02 | 31.82 | 28.36 | — | |
| Expert Divergence LearningModel Size=8B-A0.8B, Scheme=49-class2026.02 | 31.75 | 28.87 | — | |
| Expert Divergence LearningModel Size=3B-A0.3B, Scheme=3-class2026.02 | 30.78 | 28.44 | — | |
| DCLM-FastTextModel Backbone=GPT-2 XL2026.02 | — | — | 25.59 | |
| DSIRModel Backbone=GPT-2 XL2026.02 | — | — | 26.21 | |
| FineWeb-EduModel Backbone=GPT-2 XL2026.02 | — | — | 25.9 | |
| GREATSModel Backbone=GPT-2 XL2026.02 | — | — | 26.04 | |
| HyperloopTotal Parameters=1B, Active Parameters=1B, Training Tokens=100B2026.06 | — | — | 33.59 | |
| IXTModel Size=7.5B, Pre-training Dataset=Dolmino, Pre-training Strategy=from scratch2026.05 | — | — | 65.8 | |
| LoopMoETotal Parameters=3B, Active Parameters=0.8B‡, Training Tokens=200B2026.06 | — | — | 35.77 | |
| NTPModel Size=7.5B, Pre-training Dataset=Dolmino, Pre-training Strategy=from scratch2026.05 | — | — | 65.6 | |
| OLMo2-1BTotal Parameters=1.0B, Active Parameters=1.0B, Training Tokens=200B2026.06 | — | — | 33.88 | |
| OLMoE-1B-7BTotal Parameters=7B, Active Parameters=1B, Training Tokens=200B2026.06 | — | — | 35.31 | |
| OPUSModel Backbone=GPT-2 XL2026.02 | — | — | 27.5 | |
| PowerMoE-3BTotal Parameters=3B, Active Parameters=0.8B, Training Tokens=3T2026.06 | — | — | 35.79 | |
| PPLModel Backbone=GPT-2 XL2026.02 | — | — | 25.73 | |
| Pythia-1.4BTotal Parameters=1.4B, Active Parameters=1.4B, Training Tokens=200B2026.06 | — | — | 35.22 | |
| QuRatingModel Backbone=GPT-2 XL2026.02 | — | — | 23.33 | |
| RandomModel Backbone=GPT-2 XL2026.02 | — | — | 25.19 | |
| UltraFinewebModel Backbone=GPT-2 XL2026.02 | — | — | 22.58 | |
| Vanilla MoETotal Parameters=3B, Active Parameters=0.8B, Training Tokens=200B2026.06 | — | — | 35 |