ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data Cluster 2Language Modeling on Clustered Pre-training Data Cluster 2Follow18.354Perplexity (PPL)Split MoE18.3441618.4105818.47718.54342Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0318.354BaseModel Variant=BaseModel Variant=Base2026.0318.6