ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data Cluster 4Language Modeling on Clustered Pre-training Data Cluster 4Follow18.414Perplexity (PPL)Split MoE18.3091219.0170619.72520.43294Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0318.414BaseModel Variant=BaseModel Variant=Base2026.0321.036