ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data Cluster 0Language Modeling on Clustered Pre-training Data Cluster 0Follow19.819Perplexity (PPL)Split MoE19.8041619.9043320.004520.10467Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0319.819BaseModel Variant=BaseModel Variant=Base2026.0320.19