ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data Cluster 3Language Modeling on Clustered Pre-training Data Cluster 3Follow16.903Perplexity (PPL)Split MoE16.8999216.9207116.941516.96229Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0316.903BaseModel Variant=BaseModel Variant=Base2026.0316.98