Language Understanding and Reasoning on MMLU, BoolQ, ARC-e, PIQA, Hellaswag, OBQA, Winogrande (test)
28.69MMLUPre-LN + LayerNorm Scaling
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Pre-LN + LayerNorm ScalingModel Size=LLaMA-1B, Evaluation Protocol=SFT2025.02 | 28.69 | 61.8 | 48.85 | 67.92 | 33.94 | 18.6 | 54.3 | 44.87 | |
| Pre-LN + LayerNorm ScalingModel Size=LLaMA-250M, Evaluation Protocol=SFT2025.02 | 27.08 | 58.17 | 45.24 | 67.38 | 32.81 | 18.8 | 52.49 | 43.14 | |
| Pre-LNModel Size=LLaMA-1B, Evaluation Protocol=SFT2025.02 | 26.54 | 62.2 | 45.7 | 67.79 | 30.96 | 17.4 | 50.51 | 43.01 | |
| Mix-LNModel Size=LLaMA-250M, Evaluation Protocol=SFT2025.02 | 26.53 | 56.12 | 41.68 | 66.34 | 30.16 | 18 | 50.56 | 41.34 | |
| Pre-LNModel Size=LLaMA-250M, Evaluation Protocol=SFT2025.02 | 24.93 | 38.35 | 40.15 | 63.55 | 26.34 | 16.2 | 49.01 | 36.93 | |
| DeepNormModel Size=LLaMA-250M, Evaluation Protocol=SFT2025.02 | 23.6 | 37.86 | 36.62 | 61.1 | 25.69 | 15 | 49.57 | 35.63 | |
| DeepNormModel Size=LLaMA-1B, Evaluation Protocol=SFT2025.02 | 23.35 | 37.83 | 27.06 | 52.94 | 26.19 | 11.8 | 49.49 | 32.67 | |
| Mix-LNModel Size=LLaMA-1B, Evaluation Protocol=SFT2025.02 | 23.19 | 37.83 | 25.08 | 49.51 | 25.04 | 11.8 | 49.57 | 31.72 | |
| Post-LNModel Size=LLaMA-250M, Evaluation Protocol=SFT2025.02 | 22.95 | 37.83 | 26.94 | 52.72 | 26.17 | 11.6 | 49.56 | 32.54 | |
| Post-LNModel Size=LLaMA-1B, Evaluation Protocol=SFT2025.02 | 22.95 | 37.82 | 25.08 | 49.51 | 25.04 | 13.8 | 49.57 | 31.96 |