Multi-task Language Understanding on MMLU-Pro (Score, Retention)
79.7MMLU-Pro ScoreHaiku 4.5
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Haiku 4.5Training Phase=Original (Orig)2026.03 | 79.7 | — | |
| Haiku 4.5Training Phase=Trojan-Speak (TS)2026.03 | 75.6 | 95 | |
| Qwen3 32BTraining Phase=Original (Orig)2026.03 | 72.8 | — | |
| Qwen3 32BTraining Phase=Trojan-Speak (TS)2026.03 | 68.3 | 94 | |
| Qwen3 14BTraining Phase=Original (Orig)2026.03 | 67.3 | — | |
| Qwen3 8BTraining Phase=Original (Orig)2026.03 | 64.3 | — | |
| Qwen3 14BTraining Phase=Trojan-Speak (TS)2026.03 | 62.2 | 93 | |
| Qwen3 4BTraining Phase=Original (Orig)2026.03 | 58.3 | — | |
| Ouro 2.6B R4Architecture=LoopLM, # Total Params=2.6B, # Trained Tokens=7.7T2025.10 | 55.73 | — | |
| Qwen3 8BTraining Phase=Trojan-Speak (TS)2026.03 | 54.8 | 86 | |
| Qwen3Architecture=Dense, # Total Params=8.0B, # Trained Tokens=36T2025.10 | 53.72 | — | |
| Qwen3Architecture=Dense, # Total Params=4.0B, # Trained Tokens=36T2025.10 | 51.4 | — | |
| Qwen3 4BTraining Phase=Trojan-Speak (TS)2026.03 | 49.5 | 85 | |
| Gemma3Architecture=Dense, # Total Params=12.0B, # Trained Tokens=12T2025.10 | 49.21 | — | |
| Qwen2.5Architecture=Dense, # Total Params=7.0B, # Trained Tokens=18T2025.10 | 43.55 | — | |
| Llama3.1Architecture=Dense, # Total Params=8.0B, # Trained Tokens=15T2025.10 | 43.24 | — | |
| Qwen2.5Architecture=Dense, # Total Params=3.0B, # Trained Tokens=18T2025.10 | 37.87 | — | |
| Gemma3Architecture=Dense, # Total Params=4.0B, # Trained Tokens=4T2025.10 | 34.61 | — | |
| MONAModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=FewShot2026.05 | 33.75 | — | |
| Llama3.2Architecture=Dense, # Total Params=3.0B, # Trained Tokens=9T2025.10 | 33.34 | — | |
| MuonModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=FewShot2026.05 | 32.5 | — | |
| AdamWModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=FewShot2026.05 | 31.5 | — |