ResearchBenchmarksLanguage Modeling on Mini-Qwen3 (val)Language Modeling on Mini-Qwen3 (val)Follow1.763Validation LossGrouter1.723921.987712.25152.51529Feb 22, 2026Evaluation resultsMethodMethodLinksValidation LossGrouterAttn=GQA, Structure=Mo...Attn=GQA, Structure=MoE, Tokens=50B2026.021.763ZLAttn=GQA, Structure=Mo...Attn=GQA, Structure=MoE, Tokens=50B2026.022.74