ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data Cluster 5Language Modeling on Clustered Pre-training Data Cluster 5Follow19.55Perplexity (PPL)Split MoE19.5464419.5704719.594519.61853Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0319.55BaseModel Variant=BaseModel Variant=Base2026.0319.639