Language Modeling on Arxiv Proof-pile
2.35PerplexityProLong-13b
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| ProLong-13bParameters=13B, Stride=1024, Evaluation Context Window Size=163842024.05 | 2.35 | — | — | — | — | — | |
| ProLong-7bParameters=7B, Stride=1024, Evaluation Context Window Size=163842024.05 | 2.45 | — | — | — | — | — | |
| ProLong-13bParameters=13B, Stride=1024, Evaluation Context Window Size=81922024.05 | 2.47 | — | — | — | — | — | |
| Yarn-13b-64kParameters=13B, Stride=1024, Evaluation Context Window Size=163842024.05 | 2.51 | — | — | — | — | — | |
| ProLong-7bParameters=7B, Stride=1024, Evaluation Context Window Size=81922024.05 | 2.58 | — | — | — | — | — | |
| Yarn-13b-64kParameters=13B, Stride=1024, Evaluation Context Window Size=81922024.05 | 2.63 | — | — | — | — | — | |
| ProLong-13bParameters=13B, Stride=1024, Evaluation Context Window Size=40962024.05 | 2.64 | — | — | — | — | — | |
| Yarn-7b-64kParameters=7B, Stride=1024, Evaluation Context Window Size=163842024.05 | 2.66 | — | — | — | — | — | |
| Code Llama-13bParameters=13B, Stride=1024, Evaluation Context Window Size=163842024.05 | 2.67 | — | — | — | — | — | |
| Llama2-13bParameters=13B, Stride=1024, Evaluation Context Window Size=81922024.05 | 2.71 | — | — | — | — | — | |
| ProLong-7bParameters=7B, Stride=1024, Evaluation Context Window Size=40962024.05 | 2.76 | — | — | — | — | — | |
| Llama2-13bParameters=13B, Stride=1024, Evaluation Context Window Size=40962024.05 | 2.77 | — | — | — | — | — | |
| Code Llama-7bParameters=7B, Stride=1024, Evaluation Context Window Size=163842024.05 | 2.77 | — | — | — | — | — | |
| Yarn-7b-64kParameters=7B, Stride=1024, Evaluation Context Window Size=81922024.05 | 2.79 | — | — | — | — | — | |
| Yarn-13b-64kParameters=13B, Stride=1024, Evaluation Context Window Size=40962024.05 | 2.82 | — | — | — | — | — | |
| Code Llama-13bParameters=13B, Stride=1024, Evaluation Context Window Size=81922024.05 | 2.82 | — | — | — | — | — | |
| Llama2-7bParameters=7B, Stride=1024, Evaluation Context Window Size=81922024.05 | 2.88 | — | — | — | — | — | |
| ProLong-13bParameters=13B, Stride=1024, Evaluation Context Window Size=20482024.05 | 2.89 | — | — | — | — | — | |
| Llama2-7bParameters=7B, Stride=1024, Evaluation Context Window Size=40962024.05 | 2.91 | — | — | — | — | — | |
| Code Llama-7bParameters=7B, Stride=1024, Evaluation Context Window Size=81922024.05 | 2.93 | — | — | — | — | — | |
| Yarn-7b-64kParameters=7B, Stride=1024, Evaluation Context Window Size=40962024.05 | 2.99 | — | — | — | — | — | |
| ProLong-7bParameters=7B, Stride=1024, Evaluation Context Window Size=20482024.05 | 3.02 | — | — | — | — | — | |
| Llama2-13bParameters=13B, Stride=1024, Evaluation Context Window Size=20482024.05 | 3.03 | — | — | — | — | — | |
| Code Llama-13bParameters=13B, Stride=1024, Evaluation Context Window Size=40962024.05 | 3.06 | — | — | — | — | — | |
| Code Llama-13bParameters=13B, Stride=1024, Evaluation Context Window Size=20482024.05 | 3.1 | — | — | — | — | — | |
| Yarn-13b-64kParameters=13B, Stride=1024, Evaluation Context Window Size=20482024.05 | 3.1 | — | — | — | — | — | |
| Code Llama-7bParameters=7B, Stride=1024, Evaluation Context Window Size=40962024.05 | 3.17 | — | — | — | — | — | |
| Llama2-7bParameters=7B, Stride=1024, Evaluation Context Window Size=20482024.05 | 3.19 | — | — | — | — | — | |
| Yarn-7b-64kParameters=7B, Stride=1024, Evaluation Context Window Size=20482024.05 | 3.29 | — | — | — | — | — | |
| Code Llama-7bParameters=7B, Stride=1024, Evaluation Context Window Size=20482024.05 | 3.53 | — | — | — | — | — | |
| Llama2-13bParameters=13B, Stride=1024, Evaluation Context Window Size=163842024.05 | 3.58 | — | — | — | — | — | |
| Llama2-7bParameters=7B, Stride=1024, Evaluation Context Window Size=163842024.05 | 4.22 | — | — | — | — | — | |
| EntmaxContext Length=2K2025.06 | 16.85 | — | — | — | — | — | |
| ASEntmaxContext Length=2K2025.06 | 16.86 | — | — | — | — | — | |
| SSMaxContext Length=2K2025.06 | 17.02 | — | — | — | — | — | |
| SoftmaxContext Length=2K2025.06 | 17.1 | — | — | — | — | — | |
| ASEntmaxContext Length=1K2025.06 | 21.3 | — | — | — | — | — | |
| EntmaxContext Length=1K2025.06 | 21.36 | — | — | — | — | — | |
| SSMaxContext Length=1K2025.06 | 21.57 | — | — | — | — | — | |
| SoftmaxContext Length=1K2025.06 | 21.63 | — | — | — | — | — | |
| E2-LLM-16kModel Size=7B, Training Context Window=4k, Scale Parameter (G)=42024.01 | — | 2.96 | 2.71 | 2.54 | — | — | |
| E2-LLM-16kModel Size=13B, Training Context Window=4k, Scale Parameter (G)=42024.01 | — | 2.82 | 2.59 | 2.43 | — | — | |
| E2-LLM-32kModel Size=7B, Training Context Window=4k, Scale Parameter (G)=82024.01 | — | 2.99 | 2.74 | 2.56 | 2.46 | — | |
| E2-LLM-32kModel Size=13B, Training Context Window=4k, Scale Parameter (G)=82024.01 | — | 2.85 | 2.61 | 2.44 | 2.34 | — | |
| E2-LLM-64kModel Size=7B, Training Context Window=4k, Scale Parameter (G)=162024.01 | — | 3.06 | 2.81 | 2.62 | 2.51 | 2.56 | |
| E2-LLM-64kModel Size=13B, Training Context Window=4k, Scale Parameter (G)=162024.01 | — | 2.91 | 2.67 | 2.49 | 2.39 | 2.44 | |
| Llama2 (None)Model Size=7B, Training Context Window=4k2024.01 | — | 2.92 | — | — | — | — | |
| Llama2 (None)Model Size=13B, Training Context Window=4k2024.01 | — | 2.78 | — | — | — | — | |
| LongChat-v1.5-32k (PI)Model Size=7B, Training Context Window=32k2024.01 | — | 3.54 | 3.18 | 2.91 | 2.73 | — | |
| MrRoPE-ProModel=LLaMA2-7B-Instruct, Context Window=4K, Scale (S)=162026.01 | — | 5.72 | 5.39 | 4.57 | 4.11 | 3.88 | |
| MrRoPE-UniModel=LLaMA2-7B-Instruct, Context Window=4K, Scale (S)=162026.01 | — | 5.84 | 5.42 | 4.52 | 4.01 | 4.15 | |
| NTKModel=LLaMA2-7B-Instruct, Context Window=4K, Scale (S)=162026.01 | — | 6.73 | 6.76 | 6.97 | 7.11 | — | |
| Vicuna-v1.5-16k (PI)Model Size=7B, Training Context Window=16k2024.01 | — | 3.48 | 3.17 | 3.95 | — | — | |
| Vicuna-v1.5-16k (PI)Model Size=13B, Training Context Window=16k2024.01 | — | 3.27 | 2.97 | 2.78 | — | — | |
| YaRNModel=LLaMA2-7B-Instruct, Context Window=4K, Scale (S)=162026.01 | — | 6.02 | 5.59 | 4.64 | 4.07 | 4.03 |