Generation Quality and Coherence Evaluation on SlimPajama Quality Evaluation (test)
86.3Gen Quality (Std. Prefix)Self-Improving Pretraining
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Self-Improving PretrainingBackbone=Llama 1.4B, Training Dataset=SlimPajama, Pretraining Objective=Quality2026.01 | 86.3 | 50.8 | 87.9 | |
| Llama BaseBackbone=Llama 1.4B, Pretraining Objective=Base2026.01 | 50 | 47.6 | 50.1 | |
| Llama Pretrain BaselineBackbone=Llama 1.4B, Training Dataset=SlimPajama, Pretraining Objective=Next Token Prediction2026.01 | 49 | 46.8 | 49.4 |