Language Modeling on Tiny Shakespeare 32768 context length (val)
2.182LossNTK+affine
Evaluation results
| Method | Links | |
|---|---|---|
| NTK+affineTrain context length=1024, Eval context length=327682026.06 | 2.182 | |
| YaRN+affineTrain context length=1024, Eval context length=327682026.06 | 2.412 |