Generation Quality on RedPajama Generation Quality Prefixes (test)
32.4Standard Prefix CountSelf-Improving Pretraining: RF-NLL (rollout vs. rewrite)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Self-Improving Pretraining: RF-NLL (rollout vs. rewrite)Pretraining Strategy=from scratch, Model Scale=1.4B, Training Steps=21k steps, Number of rollouts=12026.01 | 32.4 | 12.1 | |
| Self-Improving Pretraining: RF-NLL (suffix vs. rewrite)Pretraining Strategy=from scratch, Model Scale=1.4B, Training Steps=21k steps, Number of rollouts=12026.01 | 5.3 | 25.8 | |
| Pretrain on RewritesPretraining Strategy=from scratch, Model Scale=1.4B, Training Steps=21k steps, Number of rollouts=12026.01 | 1.6 | 2.4 | |
| Pretrain BaselinePretraining Strategy=from scratch, Model Scale=1.4B, Training Steps=21k steps, Number of rollouts=12026.01 | 1.3 | 2.4 |