Language Understanding on MMLU-Pro (Accuracy)
72.8MMLU-Pro AccuracyFOREVER
Evaluation Results
| Method | Links | |
|---|---|---|
| FOREVERModel Scale=14B, Training Protocol=Capability-continual2026.05 | 72.8 | |
| MTLModel Scale=14B, Training Protocol=Joint-training2026.05 | 69.8 | |
| EWCModel Scale=14B, Training Protocol=Capability-continual2026.05 | 69.4 | |
| GCWMModel Scale=14B, Training Protocol=Capability-continual2026.05 | 68.8 | |
| FOREVERModel Scale=8B, Training Protocol=Capability-continual2026.05 | 68.1 | |
| Seq. SFTModel Scale=14B, Training Protocol=Capability-continual2026.05 | 67.2 | |
| AIMMergingModel Scale=8B, Training Protocol=Capability-continual2026.05 | 66.7 | |
| GCWMModel Scale=8B, Training Protocol=Capability-continual2026.05 | 66.5 | |
| OPCMModel Scale=14B, Training Protocol=Capability-continual2026.05 | 66.3 | |
| L&SModel Scale=8B, Training Protocol=Capability-continual2026.05 | 66.1 | |
| MTLModel Scale=8B, Training Protocol=Joint-training2026.05 | 65.4 | |
| MTLModel Scale=4B, Training Protocol=Joint-training2026.05 | 65.3 | |
| OPCMModel Scale=8B, Training Protocol=Capability-continual2026.05 | 65.2 | |
| OPCMModel Scale=4B, Training Protocol=Capability-continual2026.05 | 64.2 | |
| EWCModel Scale=8B, Training Protocol=Capability-continual2026.05 | 64.2 | |
| GCWMModel Scale=4B, Training Protocol=Capability-continual2026.05 | 63.9 | |
| Seq. SFTModel Scale=4B, Training Protocol=Capability-continual2026.05 | 63.6 | |
| FOREVERModel Scale=4B, Training Protocol=Capability-continual2026.05 | 62.8 | |
| Seq. SFTModel Scale=8B, Training Protocol=Capability-continual2026.05 | 60.4 | |
| EWCModel Scale=4B, Training Protocol=Capability-continual2026.05 | 60 | |
| MTLModel Scale=1.7B, Training Protocol=Joint-training2026.05 | 57.5 | |
| AIMMergingModel Scale=14B, Training Protocol=Capability-continual2026.05 | 57.5 | |
| L&SModel Scale=14B, Training Protocol=Capability-continual2026.05 | 56.8 | |
| BaseBackbone=Qwen-2.5-7B-Inst, Evaluation Protocol=0-shot2026.04 | 56.57 | |
| POPBackbone=Qwen-2.5-7B-Inst, Evaluation Protocol=0-shot2026.04 | 56.44 | |
| AIMMergingModel Scale=4B, Training Protocol=Capability-continual2026.05 | 56.4 | |
| L&SModel Scale=4B, Training Protocol=Capability-continual2026.05 | 55.6 | |
| Train on DBackbone=Qwen-2.5-7B-Inst, Evaluation Protocol=0-shot2026.04 | 53.86 | |
| GCWMModel Scale=1.7B, Training Protocol=Capability-continual2026.05 | 52 | |
| OPCMModel Scale=1.7B, Training Protocol=Capability-continual2026.05 | 51.9 | |
| ARES-SFT2026.05 | 50.56 | |
| Webscale2026.05 | 49.5 | |
| ARES-RL2026.05 | 49.36 | |
| Natural Reasoning2026.05 | 47.96 | |
| FOREVERModel Scale=1.7B, Training Protocol=Capability-continual2026.05 | 47.3 | |
| NSAModel Size=8B, Context=short-context2026.05 | 46.3 | |
| DashAttentionModel Size=8B, Context=short-context2026.05 | 46.2 | |
| FullAttnModel Size=8B, Context=short-context2026.05 | 46.1 | |
| CPT2026.05 | 46.02 | |
| InfLLMv2Model Size=8B, Context=short-context2026.05 | 45.6 | |
| MTLModel Scale=0.6B, Training Protocol=Joint-training2026.05 | 45.1 | |
| AIMMergingModel Scale=1.7B, Training Protocol=Capability-continual2026.05 | 44.3 | |
| POPBackbone=Qwen-2.5-7B, Evaluation Protocol=0-shot2026.04 | 43.82 | |
| L&SModel Scale=1.7B, Training Protocol=Capability-continual2026.05 | 43.5 | |
| EWCModel Scale=1.7B, Training Protocol=Capability-continual2026.05 | 40.5 | |
| Train on DBackbone=Qwen-2.5-7B, Evaluation Protocol=0-shot2026.04 | 38.07 | |
| Seq. SFTModel Scale=1.7B, Training Protocol=Capability-continual2026.05 | 35.3 | |
| BaseBackbone=Qwen-2.5-7B, Evaluation Protocol=0-shot2026.04 | 30.1 | |
| AIMMergingModel Scale=0.6B, Training Protocol=Capability-continual2026.05 | 29.4 | |
| FOREVERModel Scale=0.6B, Training Protocol=Capability-continual2026.05 | 29.1 | |
| L&SModel Scale=0.6B, Training Protocol=Capability-continual2026.05 | 29 | |
| GCWMModel Scale=0.6B, Training Protocol=Capability-continual2026.05 | 27.8 | |
| OPCMModel Scale=0.6B, Training Protocol=Capability-continual2026.05 | 26.7 | |
| EWCModel Scale=0.6B, Training Protocol=Capability-continual2026.05 | 26.6 | |
| Seq. SFTModel Scale=0.6B, Training Protocol=Capability-continual2026.05 | 24.2 | |
| LFM2.5 350MTraining tokens=28T, Evaluation Protocol=0-shot2026.04 | 18.6 | |
| Nautile-370MTraining tokens=∼0.8T, Evaluation Protocol=0-shot2026.04 | 14.9 | |
| Qwen2.5 0.5BTraining tokens=18T, Evaluation Protocol=0-shot2026.04 | 14.3 | |
| Granite 350MTraining tokens=10–12T, Evaluation Protocol=0-shot2026.04 | 11.2 | |
| SmolLM2 360MTraining tokens=4T, Evaluation Protocol=0-shot2026.04 | 10.3 |