ResearchBenchmarksLanguage Modeling on dolma_pes2o (val)Language Modeling on dolma_pes2o (val)Follow11.301PerplexityHi-MoE-7B11.2856411.3893211.49311.59668May 8, 2026Evaluation resultsMethodMethodLinksPerplexityRelative Change (%)Hi-MoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=8, grouping=4 hierarchical groups, experts per group=22026.0511.301-3.3OLMoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=82026.0511.685—