Language Modeling on LLaMA-2 Pre-training (val)
28.52Perplexity (1.3B tokens)6-bit/8-bit
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 6-bit/8-bitprecision=6-bit/8-bit, αt=1.5/1.1, βt=1.52026.05 | 28.52 | 21.17 | 19.07 | 17.88 | 17.06 | 16.43 | 15.93 | 15.57 | 15.2 | |
| 16-bitprecision=16-bit, αt=N.A., βt=N.A.2026.05 | 29.1 | 21.35 | 19.08 | 17.84 | 16.98 | 16.33 | 15.82 | 15.44 | 15.06 | |
| 8-bit/16-bitprecision=8-bit/16-bit, αt=1.5/1.1, βt=N.A.2026.05 | 29.5 | 21.48 | 19.14 | 17.89 | 17.04 | 16.37 | 15.86 | 15.48 | 15.1 | |
| 8-bitprecision=8-bit, αt=N.A., βt=N.A.2026.05 | 29.54 | 21.57 | 19.25 | 17.96 | 17.11 | 16.45 | 15.94 | 15.56 | 15.18 |