Language Modeling on OpenWebText (test)
2.947Average PerplexityHi-MoE
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Hi-MoEBase Model=nanoGPT, Total Parameters=1.06B, Active Parameters per Token=507M2026.05 | 2.947 | — | 23 | |
| MoGEBase Model=nanoGPT, Total Parameters=1.06B, Active Parameters per Token=507M2026.05 | 2.977 | — | 25 | |
| Loss-free balancingBase Model=nanoGPT, Total Parameters=1.06B, Active Parameters per Token=507M2026.05 | 2.979 | — | 37 | |
| ST-MoEBase Model=nanoGPT, Total Parameters=1.06B, Active Parameters per Token=507M2026.05 | 2.981 | — | 33 | |
| Vanilla GShard-styleBase Model=nanoGPT, Total Parameters=1.06B, Active Parameters per Token=507M2026.05 | 2.985 | — | 31 | |
| Superposed DecodingModel=Mistral 7B, Evaluation Model=Llama-2-70B, Draft #=Best2024.05 | 10.87 | — | — | |
| Superposed DecodingModel=Mistral 7B, Evaluation Model=Llama-2-70B, Draft #=12024.05 | 11.34 | — | — | |
| Nucleus SamplingModel=Mistral 7B, Evaluation Model=Llama-2-70B2024.05 | 11.42 | — | — | |
| Superposed DecodingModel=Mistral 7B, Evaluation Model=Llama-2-70B, Draft #=22024.05 | 12.74 | — | — | |
| Superposed DecodingModel=Mistral 7B, Evaluation Model=Llama-2-70B, Draft #=32024.05 | 13.63 | — | — | |
| ARTraining tokens=262B2024.06 | 17.54 | — | — | |
| AR2024.10 | 17.56 | — | — | |
| EDLM-coAR2024.10 | 17.58 | — | — | |
| H3 Hybrid (125M)Params=~125M, Tokens Seen=~50B2026.06 | 19.6 | — | — | |
| PHA-125MParams=125M, Tokens Seen=~12B2026.06 | 19.72 | — | — | |
| GSS HybridParams=~125M, Tokens Seen=~50B2026.06 | 19.8 | — | — | |
| EDLM-AR2024.10 | 20.49 | — | — | |
| TransformerParams=~125M, Tokens Seen=~50B2026.06 | 20.6 | — | — | |
| H3Params=~125M, Tokens Seen=~50B2026.06 | 21 | — | — | |
| EDLM-NCE2024.10 | 21.52 | — | — | |
| MDLMTraining tokens=262B2024.06 | 23.21 | — | — | |
| MLDM2024.10 | 23.83 | — | — | |
| GSSParams=~125M, Tokens Seen=~50B2026.06 | 24 | — | — | |
| SEDDTraining tokens=262B, Retrained=true2024.06 | 24.1 | — | — | |
| SEDD2024.10 | 24.56 | — | — | |
| DropoutData Removal Ratio=10%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 25.51 | — | — | |
| Random RankingData Removal Ratio=10%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 25.52 | — | — | |
| LOGRAData Removal Ratio=10%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 25.68 | — | — | |
| LOGRAData Removal Ratio=20%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 25.89 | — | — | |
| Random RankingData Removal Ratio=20%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 25.91 | — | — | |
| DropoutData Removal Ratio=20%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 26.05 | — | — | |
| Random RankingData Removal Ratio=35%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 26.28 | — | — | |
| Random RankingData Removal Ratio=40%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 26.49 | — | — | |
| DropoutData Removal Ratio=35%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 26.55 | — | — | |
| LOGRAData Removal Ratio=35%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 26.62 | — | — | |
| LOGRAData Removal Ratio=40%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 26.79 | — | — | |
| DropoutData Removal Ratio=40%, Backbone=Pythia 1.4B, Averaged over=5 runs2025.09 | 26.8 | — | — | |
| Subspace Networks (Decentralized Compressed)B/W=80Mbps, TPS=592.412025.06 | 46.75 | — | — | |
| CentralizedB/W=100Gbps, TPS=602.572025.06 | 47.22 | — | — | |
| DecentralizedB/W=80Mbps, TPS=36.122025.06 | 925.19 | — | — | |
| Coupled AdamDataset Size (D)=5B, Model Size (N)=125M2025.02 | — | 3.12 | — | |
| Coupled AdamDataset Size (D)=5B, Model Size (N)=355M2025.02 | — | 2.93 | — | |
| Coupled AdamDataset Size (D)=5B, Model Size (N)=760M2025.02 | — | 2.86 | — | |
| Coupled AdamDataset Size (D)=10B, Model Size (N)=125M2025.02 | — | 3.03 | — | |
| Coupled AdamDataset Size (D)=10B, Model Size (N)=355M2025.02 | — | 2.83 | — | |
| Coupled AdamDataset Size (D)=10B, Model Size (N)=760M2025.02 | — | 2.74 | — | |
| Coupled AdamDataset Size (D)=20B, Model Size (N)=125M2025.02 | — | 2.97 | — | |
| Coupled AdamDataset Size (D)=20B, Model Size (N)=355M2025.02 | — | 2.75 | — | |
| Coupled AdamDataset Size (D)=20B, Model Size (N)=760M2025.02 | — | 2.65 | — | |
| Standard AdamDataset Size (D)=5B, Model Size (N)=125M2025.02 | — | 3.14 | — | |
| Standard AdamDataset Size (D)=5B, Model Size (N)=355M2025.02 | — | 2.95 | — | |
| Standard AdamDataset Size (D)=5B, Model Size (N)=760M2025.02 | — | 2.85 | — | |
| Standard AdamDataset Size (D)=10B, Model Size (N)=125M2025.02 | — | 3.07 | — | |
| Standard AdamDataset Size (D)=10B, Model Size (N)=355M2025.02 | — | 2.86 | — | |
| Standard AdamDataset Size (D)=10B, Model Size (N)=760M2025.02 | — | 2.75 | — | |
| Standard AdamDataset Size (D)=20B, Model Size (N)=125M2025.02 | — | 3.03 | — | |
| Standard AdamDataset Size (D)=20B, Model Size (N)=355M2025.02 | — | 2.79 | — | |
| Standard AdamDataset Size (D)=20B, Model Size (N)=760M2025.02 | — | 2.68 | — |