List your company
ResearchFormerly WizWand

Language Modeling on The Pile (val) (Loss and PPL)

4.24PPLAC-ODM
3.82846.60679.38512.1633May 29, 2025Aug 2, 2025Oct 6, 2025Dec 10, 2025Feb 13, 2026Apr 19, 2026Jun 23, 2026
Updated 4mo ago

Evaluation results

MethodLinks
AC-ODMTraining Steps=20,832, Target Model=Pythia-12B, Proxy Model=Pythia-1B2025.054.24—
HDSSteps=20,832, Model=Pythia-12B2026.064.89—
UNIPOOLScale=830M, Arch=48/1024, MoE configuration=8E/top-12026.055.4321.6923
UNIPOOLScale=978M, Arch=24/1536, MoE configuration=8E/top-12026.055.47361.6999
Vanilla MoEScale=978M, Arch=24/1536, MoE configuration=8E/top-12026.055.56831.7171
UNIPOOLScale=650M, Arch=36/1024, MoE configuration=8E/top-12026.055.61861.726
Vanilla MoEScale=830M, Arch=48/1024, MoE configuration=8E/top-12026.055.64581.7309
Vanilla MoEScale=650M, Arch=36/1024, MoE configuration=8E/top-12026.055.7941.7568
UNIPOOLScale=469M, Arch=24/1024, MoE configuration=8E/top-12026.055.83341.7636
AC-ODMTraining Steps=15,624, Target Model=Pythia-12B, Proxy Model=Pythia-1B2025.055.91—
Vanilla MoEScale=469M, Arch=24/1024, MoE configuration=8E/top-12026.056.03881.7982
HDSSteps=15,624, Model=Pythia-12B2026.066.04—
DenseScale=830M, Arch=48/1024, MoE configuration=None (Dense)2026.056.06941.8032
DenseScale=978M, Arch=24/1536, MoE configuration=None (Dense)2026.056.1841.822
DenseScale=650M, Arch=36/1024, MoE configuration=None (Dense)2026.056.24531.8318
DenseScale=469M, Arch=24/1024, MoE configuration=None (Dense)2026.056.5931.886
UNIPOOLScale=182M, Arch=12/768, MoE configuration=8E/top-12026.056.70581.9029
Vanilla MoEScale=182M, Arch=12/768, MoE configuration=8E/top-12026.056.90121.9317
Depth-AttentionSize=3B, Evaluation Protocol=Zero-shot2026.067.25—
Attention ResidualsSize=3B, Evaluation Protocol=Zero-shot2026.067.27—
mHCSize=3B, Evaluation Protocol=Zero-shot2026.067.3—
ODMTraining Steps=20,832, Target Model=Pythia-12B2025.057.32—
ODMSteps=20,832, Model=Pythia-12B2026.067.32—
DenseFormerSize=3B, Evaluation Protocol=Zero-shot2026.067.35—
VanillaSize=3B, Evaluation Protocol=Zero-shot2026.067.55—
AC-ODMTraining Steps=10,416, Target Model=Pythia-12B, Proxy Model=Pythia-1B2025.057.56—
DenseScale=182M, Arch=12/768, MoE configuration=None (Dense)2026.057.7082.042
Depth-AttentionSize=1.5B, Evaluation Protocol=Zero-shot2026.067.77—
Attention ResidualsSize=1.5B, Evaluation Protocol=Zero-shot2026.067.78—
mHCSize=1.5B, Evaluation Protocol=Zero-shot2026.067.88—
DenseFormerSize=1.5B, Evaluation Protocol=Zero-shot2026.067.88—
VanillaSize=1.5B, Evaluation Protocol=Zero-shot2026.068.17—
HDSSteps=10,416, Model=Pythia-12B2026.068.19—
ODMTraining Steps=15,624, Target Model=Pythia-12B2025.058.55—
ODMSteps=15,624, Model=Pythia-12B2026.068.55—
ODMTraining Steps=10,416, Target Model=Pythia-12B2025.0510.01—
ODMSteps=10,416, Model=Pythia-12B2026.0610.01—
AC-ODMTraining Steps=5,208, Target Model=Pythia-12B, Proxy Model=Pythia-1B2025.0512.87—
HDSSteps=5,208, Model=Pythia-12B2026.0613.46—
ODMTraining Steps=5,208, Target Model=Pythia-12B2025.0514.53—
ODMSteps=5,208, Model=Pythia-12B2026.0614.53—