ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data Cluster 15Language Modeling on Clustered Pre-training Data Cluster 15Follow18.294Perplexity (PPL)Split MoE18.2901618.3160818.34218.36792Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0318.294BaseModel Variant=BaseModel Variant=Base2026.0318.39