ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data Cluster 14Language Modeling on Clustered Pre-training Data Cluster 14Follow20.656Perplexity (PPL)Split MoE20.6508820.6854420.7220.75456Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0320.656BaseModel Variant=BaseModel Variant=Base2026.0320.784