ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data Cluster 13Language Modeling on Clustered Pre-training Data Cluster 13Follow19.461Perplexity (PPL)Split MoE19.4576819.4800919.502519.52491Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0319.461BaseModel Variant=BaseModel Variant=Base2026.0319.544