ResearchBenchmarksLanguage Modeling on dolma_stack (val)Follow3.765PerplexityHi-MoE-7B3.763163.775583.7883.80042May 8, 2026Evaluation ResultsMethodMethodLinksPerplexityRelative Change (%)Hi-MoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=8, grouping=4 hierarchical groups, experts per group=22026.053.765-1.2OLMoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=82026.053.811—