Language Modeling on Evaluation Set
1.844LossConceptMoE-A1B-24B
Evaluation results
| Method | Links | |
|---|---|---|
| ConceptMoE-A1B-24BTokens=559B2026.01 | 1.844 | |
| MoE-A1B-24BTokens=559B2026.01 | 1.851 | |
| ConceptMoE-A0.5B-12BTokens=243B2026.01 | 1.99 | |
| MoE-A0.5B-12BTokens=243B2026.01 | 1.992 |