ResearchBenchmarksLanguage Modeling on dolma books (val)For AI assistantsMarkdownContent indexLanguage Modeling on dolma books (val)Follow36.674PerplexityHi-MoE-7B36.552837.370938.18939.0071May 8, 2026Evaluation resultsMethodMethodLinksPerplexityRelative Change (%)Hi-MoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=8, grouping=4 hierarchical groups, experts per group=22026.0536.674-7.6OLMoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=82026.0539.704—