ResearchBenchmarksLanguage Modeling on ice (val)Language Modeling on ice (val)Follow23.819PerplexityOLMoE-7B23.8162423.8348723.853523.87213May 8, 2026Evaluation resultsMethodMethodLinksPerplexityDelta (%)OLMoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=82026.0523.819—Hi-MoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=8, grouping=4 hierarchical groups, experts per group=22026.0523.88830