Language Modeling on The Pile (val) (Loss and PPL)
4.24PPLAC-ODM
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| AC-ODMTraining Steps=20,832, Target Model=Pythia-12B, Proxy Model=Pythia-1B2025.05 | 4.24 | — | |
| HDSSteps=20,832, Model=Pythia-12B2026.06 | 4.89 | — | |
| UNIPOOLScale=830M, Arch=48/1024, MoE configuration=8E/top-12026.05 | 5.432 | 1.6923 | |
| UNIPOOLScale=978M, Arch=24/1536, MoE configuration=8E/top-12026.05 | 5.4736 | 1.6999 | |
| Vanilla MoEScale=978M, Arch=24/1536, MoE configuration=8E/top-12026.05 | 5.5683 | 1.7171 | |
| UNIPOOLScale=650M, Arch=36/1024, MoE configuration=8E/top-12026.05 | 5.6186 | 1.726 | |
| Vanilla MoEScale=830M, Arch=48/1024, MoE configuration=8E/top-12026.05 | 5.6458 | 1.7309 | |
| Vanilla MoEScale=650M, Arch=36/1024, MoE configuration=8E/top-12026.05 | 5.794 | 1.7568 | |
| UNIPOOLScale=469M, Arch=24/1024, MoE configuration=8E/top-12026.05 | 5.8334 | 1.7636 | |
| AC-ODMTraining Steps=15,624, Target Model=Pythia-12B, Proxy Model=Pythia-1B2025.05 | 5.91 | — | |
| Vanilla MoEScale=469M, Arch=24/1024, MoE configuration=8E/top-12026.05 | 6.0388 | 1.7982 | |
| HDSSteps=15,624, Model=Pythia-12B2026.06 | 6.04 | — | |
| DenseScale=830M, Arch=48/1024, MoE configuration=None (Dense)2026.05 | 6.0694 | 1.8032 | |
| DenseScale=978M, Arch=24/1536, MoE configuration=None (Dense)2026.05 | 6.184 | 1.822 | |
| DenseScale=650M, Arch=36/1024, MoE configuration=None (Dense)2026.05 | 6.2453 | 1.8318 | |
| DenseScale=469M, Arch=24/1024, MoE configuration=None (Dense)2026.05 | 6.593 | 1.886 | |
| UNIPOOLScale=182M, Arch=12/768, MoE configuration=8E/top-12026.05 | 6.7058 | 1.9029 | |
| Vanilla MoEScale=182M, Arch=12/768, MoE configuration=8E/top-12026.05 | 6.9012 | 1.9317 | |
| Depth-AttentionSize=3B, Evaluation Protocol=Zero-shot2026.06 | 7.25 | — | |
| Attention ResidualsSize=3B, Evaluation Protocol=Zero-shot2026.06 | 7.27 | — | |
| mHCSize=3B, Evaluation Protocol=Zero-shot2026.06 | 7.3 | — | |
| ODMTraining Steps=20,832, Target Model=Pythia-12B2025.05 | 7.32 | — | |
| ODMSteps=20,832, Model=Pythia-12B2026.06 | 7.32 | — | |
| DenseFormerSize=3B, Evaluation Protocol=Zero-shot2026.06 | 7.35 | — | |
| VanillaSize=3B, Evaluation Protocol=Zero-shot2026.06 | 7.55 | — | |
| AC-ODMTraining Steps=10,416, Target Model=Pythia-12B, Proxy Model=Pythia-1B2025.05 | 7.56 | — | |
| DenseScale=182M, Arch=12/768, MoE configuration=None (Dense)2026.05 | 7.708 | 2.042 | |
| Depth-AttentionSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 7.77 | — | |
| Attention ResidualsSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 7.78 | — | |
| mHCSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 7.88 | — | |
| DenseFormerSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 7.88 | — | |
| VanillaSize=1.5B, Evaluation Protocol=Zero-shot2026.06 | 8.17 | — | |
| HDSSteps=10,416, Model=Pythia-12B2026.06 | 8.19 | — | |
| ODMTraining Steps=15,624, Target Model=Pythia-12B2025.05 | 8.55 | — | |
| ODMSteps=15,624, Model=Pythia-12B2026.06 | 8.55 | — | |
| ODMTraining Steps=10,416, Target Model=Pythia-12B2025.05 | 10.01 | — | |
| ODMSteps=10,416, Model=Pythia-12B2026.06 | 10.01 | — | |
| AC-ODMTraining Steps=5,208, Target Model=Pythia-12B, Proxy Model=Pythia-1B2025.05 | 12.87 | — | |
| HDSSteps=5,208, Model=Pythia-12B2026.06 | 13.46 | — | |
| ODMTraining Steps=5,208, Target Model=Pythia-12B2025.05 | 14.53 | — | |
| ODMSteps=5,208, Model=Pythia-12B2026.06 | 14.53 | — |