Safety Evaluation on RedPajama Safety Evals (test)
93.4Safety Score (Avg)Self-Improving Pretraining
Evaluation results
| Method | Links | |
|---|---|---|
| Self-Improving PretrainingPre-training Data=RedPajama, Pre-training Strategy=Self-Improving2026.01 | 93.4 | |
| Llama BasePre-training Strategy=Standard next token prediction2026.01 | 68 | |
| Llama Pretrain BaselinePre-training Data=RedPajama, Pre-training Strategy=Standard next token prediction2026.01 | 67.4 | |
| Self-Improving Pretraining: RF-NLL (rollout vs. rewrite)Pretraining Strategy=from scratch, Model Scale=1.4B, Training Steps=21k steps, Number of rollouts=12026.01 | 0.975 | |
| Pretrain on RewritesPretraining Strategy=from scratch, Model Scale=1.4B, Training Steps=21k steps, Number of rollouts=12026.01 | 0.967 | |
| Self-Improving Pretraining: RF-NLL (suffix vs. rewrite)Pretraining Strategy=from scratch, Model Scale=1.4B, Training Steps=21k steps, Number of rollouts=12026.01 | 0.964 | |
| Pretrain BaselinePretraining Strategy=from scratch, Model Scale=1.4B, Training Steps=21k steps, Number of rollouts=12026.01 | 0.852 |