ResearchBenchmarksLanguage Modeling on dolma common-crawl (val)Language Modeling on dolma common-crawl (val)Follow39.872PerplexityHi-MoE-7B39.75840.527541.29742.0665May 8, 2026Evaluation resultsMethodMethodLinksPerplexityRelative Change (%)Hi-MoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=8, grouping=4 hierarchical groups, experts per group=22026.0539.872-6.7OLMoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=82026.0542.722—