Language Modeling on TinyStories 60M tokens (val)
51.23PPL (Val)TernaryLM (Full precision)
Evaluation Results
| Method | Links | |
|---|---|---|
| TernaryLM (Full precision)Params=132M, Precision=FP322026.02 | 51.23 | |
| GPT-2 SmallParams=124M, Precision=FP322026.02 | 52.31 | |
| TinyLLaMA-100MParams=100M, Precision=FP162026.02 | 54.12 | |
| BERT-Base (adapted)Params=110M, Precision=FP322026.02 | 55.87 | |
| TernaryLMParams=132M, Precision=1-bit2026.02 | 58.42 | |
| TinyStories-33MParams=33M, Precision=FP162026.02 | 62.4 | |
| GPT-2 SmallParams=124M, Precision=FP162026.02 | 65 | |
| TinyStories-3M*Params=3M, Precision=FP162026.02 | 86.5 |