Large Language Model Evaluation on AlpacaEval, TruthfulQA, and MMLU (test)
78.1AlpacaEval ScoreWarmup-Stable-Only (WSO)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Warmup-Stable-Only (WSO)Model=1B, Scheduler=Warmup-Stable-Only (WSO), alpha_pre=1.0, Pre-training tokens=2T, Selected SFT LR=1 × 10−42026.03 | 78.1 | 38.7 | 34.5 | 50.4 | |
| WSDModel=1B, Scheduler=WSD, alpha_pre=0.1, Pre-training tokens=2T, Selected SFT LR=3 × 10−52026.03 | 77.2 | 38.3 | 33.6 | 49.7 | |
| CosineModel=1B, Scheduler=Cosine, alpha_pre=0.0, Pre-training tokens=2T, Selected SFT LR=1 × 10−42026.03 | 76.4 | 37.9 | 33.9 | 49.4 | |
| WSDModel=1B, Scheduler=WSD, alpha_pre=0.0, Pre-training tokens=2T, Selected SFT LR=3 × 10−52026.03 | 76 | 38.4 | 33.7 | 49.4 | |
| CosineModel=1B, Scheduler=Cosine, alpha_pre=0.1, Pre-training tokens=2T, Selected SFT LR=1 × 10−52026.03 | 76 | 37.9 | 33.9 | 49.3 | |
| LinearModel=1B, Scheduler=Linear, alpha_pre=0.1, Pre-training tokens=2T, Selected SFT LR=3 × 10−52026.03 | 75.6 | 37.8 | 34.2 | 49.2 | |
| LinearModel=1B, Scheduler=Linear, alpha_pre=0.0, Pre-training tokens=2T, Selected SFT LR=3 × 10−52026.03 | 75.5 | 37.9 | 33.9 | 49.1 |