Question Answering on MMLU STEM (test)
0.0116LossDelayed scaling
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Delayed scalingBackbone=Llama-2-13B, Fine-tuning steps=3000, Learning rate=1e-5, Batch size=4, Sequence length=10242026.02 | 0.0116 | 68 | 70.7 | 33 | |
| Auto-alpha spectral scalingBackbone=Llama-2-13B, Fine-tuning steps=3000, Learning rate=1e-5, Batch size=4, Sequence length=1024, burn-in=100-step, kappa=12026.02 | 0.0116 | 0 | 31.2 | 33.6 | |
| Conservative spectral scalingBackbone=Llama-2-13B, Fine-tuning steps=3000, Learning rate=1e-5, Batch size=4, Sequence length=1024, alpha=0.032026.02 | 0.0119 | 0 | 0.5 | 28.1 |