ResearchBenchmarksLanguage Modeling on Clustered Pre-training Data Cluster 12Language Modeling on Clustered Pre-training Data Cluster 12Follow17.731Perplexity (PPL)Split MoE17.7275217.7510117.774517.79799Mar 19, 2026Evaluation resultsMethodMethodLinksPerplexity (PPL)Split MoEModel Variant=Split MoEModel Variant=Split MoE2026.0317.731BaseModel Variant=BaseModel Variant=Base2026.0317.818