ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data AverageLanguage Modeling on Clustered Pre-training Data AverageFollow18.008Perplexity (PPL)Split MoE17.9920418.0997718.207518.31523Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0318.008BaseModel Variant=BaseModel Variant=Base2026.0318.407