Language Model Pre-training on Datacomp-LM 400M-1x scale 1.0
17.9Core ScoreDCLM-baseline + SIEVE
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| DCLM-baseline + SIEVETraining Tokens=8.2B, Model Parameters=400M, Teacher Model=GPT-4o2024.10 | 17.9 | 10.08 | 25.99 | |
| DCLM-baseline (reproduced)Training Tokens=8.2B, Model Parameters=400M, Filtering=fastText-based2024.10 | 17.61 | 9.46 | 23.89 | |
| Fineweb-edu (reproduced)Training Tokens=8.2B, Model Parameters=400M, Filtering=Llama3-annotated data2024.10 | 16.85 | 8.46 | 23.32 |