ResearchBenchmarksLanguage Modeling on m2d2 s2orc (val)Follow21.803PerplexityHi-MoE-7B21.7436822.1440922.544522.94491May 8, 2026Evaluation ResultsMethodMethodLinksPerplexityRelative Change (%)Hi-MoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=8, grouping=4 hierarchical groups, experts per group=22026.0521.803-6.4OLMoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=82026.0523.286—