Language Modeling on Proof-pile 128 32k (test)
2.28PerplexityProLong-13b
Evaluation Results
| Method | Links | |
|---|---|---|
| ProLong-13bParameters=13B, Stride=1024, Evaluation Context Window Size=327682024.05 | 2.28 | |
| ProLong-7bParameters=7B, Stride=1024, Evaluation Context Window Size=327682024.05 | 2.38 | |
| Yarn-13b-64kParameters=13B, Stride=1024, Evaluation Context Window Size=327682024.05 | 2.43 | |
| Yarn-7b-64kParameters=7B, Stride=1024, Evaluation Context Window Size=327682024.05 | 2.6 | |
| Code Llama-13bParameters=13B, Stride=1024, Evaluation Context Window Size=327682024.05 | 2.71 | |
| Code Llama-7bParameters=7B, Stride=1024, Evaluation Context Window Size=327682024.05 | 2.84 | |
| Llama2-13bParameters=13B, Stride=1024, Evaluation Context Window Size=327682024.05 | 14.69 | |
| Llama2-7bParameters=7B, Stride=1024, Evaluation Context Window Size=327682024.05 | 18.72 |