Language Modeling on FineWeb-Edu 100B (val)
2.62CE LossET
Evaluation results
| Method | Links | |
|---|---|---|
| ETBatch=256k → 500M, EMA decay (β)=0.999, warmup=true2026.03 | 2.62 | |
| ECBatch=256k2026.03 | 2.621 | |
| TC auxBatch=32k2026.03 | 2.687 | |
| denseBatch=—2026.03 | 2.751 | |
| ECBatch=64k2026.03 | 2.841 | |
| ECBatch=512k2026.03 | 2.843 | |
| ETBatch=0.5M → 500M, Hyperparameter=β=0.999+warmup2026.03 | 2.844 | |
| ECBatch=8k2026.03 | 2.845 | |
| TC auxBatch=64k2026.03 | 2.892 | |
| TCBatch=—2026.03 | 2.893 | |
| TC loss-freeBatch=512k2026.03 | 2.898 | |
| ECBatch=2k2026.03 | 2.91 | |
| denseBatch=—2026.03 | 3.002 |