Language Modeling on WikiText-2 32768 context length (val)
2.048LossNTK+affine
Evaluation results
| Method | Links | |
|---|---|---|
| NTK+affineTrain context length=1024, Eval context length=327682026.06 | 2.048 | |
| YaRN+affineTrain context length=1024, Eval context length=327682026.06 | 2.267 |