ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data Cluster 10Language Modeling on Clustered Pre-training Data Cluster 10Follow17.783Perplexity (PPL)Split MoE17.7722417.8448717.917517.99013Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0317.783BaseModel Variant=BaseModel Variant=Base2026.0318.052