ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data Cluster 7Language Modeling on Clustered Pre-training Data Cluster 7Follow15.972Perplexity (PPL)Split MoE15.9399216.1564616.37316.58954Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0315.972BaseModel Variant=BaseModel Variant=Base2026.0316.774