ResearchBenchmarksLanguage Modeling on Pile (val) (Perplexity and Delta)Follow12.978PerplexityHi-MoE-7B12.9625613.0667813.17113.27522May 8, 2026Evaluation ResultsMethodMethodLinksPerplexityDelta (%)Hi-MoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=8, grouping=4 hierarchical groups, experts per group=22026.0512.978-2.9OLMoE-7Btransformer layers=16,...transformer layers=16, hidden size=2048, experts=64, routing=top-k=82026.0513.364—