Language Modeling on FineWeb-Edu (held-out)
4.733Final 50 Train LossAdamW
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| AdamWTrain Data=Clean, Variant=LOO+Jensen BC, Batch / Split=b512, LOO-64 embed, LR=6e−4; dense 9e−42026.05 | 4.733 | 4.687 | -0.149 | |
| AdamWTrain Data=Clean, Variant=Standard, Batch / Split=b512, LR=6e−42026.05 | 4.832 | 4.836 | — | |
| AdamWTrain Data=Noisy, Variant=Standard, Batch / Split=b512, LR=6e−42026.05 | 4.853 | 4.823 | — | |
| AdamWTrain Data=Noisy, Variant=LOO+Jensen BC, Batch / Split=b512, LOO-64 embed, LR=6e−4; dense 9e−42026.05 | 4.872 | 4.803 | -0.019 | |
| ShampooTrain Data=Clean, Variant=Full BC, Batch / Split=A512/B512, LR=1e−32026.05 | 5.676 | 5.681 | -0.11 | |
| ShampooTrain Data=Clean, Variant=Standard, Batch / Split=b512, LR=6e−42026.05 | 5.787 | 5.792 | — | |
| SophiaTrain Data=Clean, Variant=Full BC, Batch / Split=A512/B512, LR=6e−42026.05 | 6.592 | 6.595 | -0.07 | |
| SophiaTrain Data=Clean, Variant=Standard, Batch / Split=b512, LR=6e−42026.05 | 6.663 | 6.665 | — |