Language Modeling on Proof-pile
2.33PerplexitySHAREDLLM
Evaluation results
| Method | Links | |
|---|---|---|
| SHAREDLLMContext Length=4K2026.03 | 2.33 | |
| SHAREDLLMContext Length=8K2026.03 | 2.34 | |
| CEPEContext Length=4K2026.03 | 2.38 | |
| SHAREDLLMContext Length=32K2026.03 | 2.38 | |
| CEPEContext Length=128K2026.03 | 2.39 | |
| SHAREDLLMContext Length=128K2026.03 | 2.4 | |
| CEPEContext Length=8K2026.03 | 2.43 | |
| CEPEContext Length=32K2026.03 | 2.45 | |
| LLaMA-2-32KContext Length=32K2026.03 | 2.47 | |
| YaRNContext Length=32K2026.03 | 2.47 | |
| LLaMA-3.1Context Length=128K2026.03 | 2.48 | |
| PIContext Length=32K2026.03 | 2.51 | |
| LLaMA-3.1Context Length=32K2026.03 | 2.52 | |
| LLaMA-3.1Context Length=8K2026.03 | 2.54 | |
| YaRNContext Length=8K2026.03 | 2.56 | |
| LLaMA-2-32KContext Length=8K2026.03 | 2.58 | |
| LLaMA-3.1Context Length=4K2026.03 | 2.58 | |
| LongLoRA-7B-32kModel Size=7B, Context Length=16k2025.03 | 2.6 | |
| LongLoRA-7B-32kContext Length=16k, Model Size=7B2026.06 | 2.6 | |
| PIContext Length=8K2026.03 | 2.64 | |
| OpenLLaMA-3BModel Size=3B, Context Length=2k2025.03 | 2.7 | |
| OpenLLaMA-3BContext Length=2k, Model Size=3B2026.06 | 2.7 | |
| LLaMA-2-7BModel Size=7B, Context Length=4k2025.03 | 2.72 | |
| LLaMA-2-7BContext Length=4k, Model Size=7B2026.06 | 2.72 | |
| PIContext Length=4K2026.03 | 2.77 | |
| Phi3-128kModel Size=3B, Context Length=4k2025.03 | 2.77 | |
| Phi3-128kContext Length=4k, Model Size=3B2026.06 | 2.77 | |
| YaRNContext Length=4K2026.03 | 2.82 | |
| LLaMA-2-32KContext Length=4K2026.03 | 2.87 | |
| LongLLaMA-3B†Model Size=3B, Context Length=4k2025.03 | 2.94 | |
| LongLLaMA-3BContext Length=4k, Model Size=3B, Memory=Extended Memory2026.06 | 2.94 | |
| OpenLLaMA-3BModel Size=3B, Context Length=1k2025.03 | 2.96 | |
| OpenLLaMA-3BContext Length=1k, Model Size=3B2026.06 | 2.96 | |
| YARN-128k-7bModel Size=7B, Context Length=4k2025.03 | 2.98 | |
| ERMAR (w/ 4K capacity)Model Size=3B, Context Length=2k, Memory Capacity=4K2025.03 | 2.98 | |
| YARN-128k-7BContext Length=4k, Model Size=7B2026.06 | 2.98 | |
| ERMARContext Length=2k, Model Size=3B, Memory=4K2026.06 | 2.98 | |
| MemLong (w/ 4K capacity)Model Size=3B, Context Length=16k, Memory Capacity=4K2025.03 | 2.99 | |
| MemLongContext Length=16k, Model Size=3B, Memory=4K2026.06 | 2.99 | |
| YARN-128k-7bModel Size=7B, Context Length=1k2025.03 | 3.03 | |
| ERMAR (w/ 4K capacity)Model Size=3B, Context Length=4k, Memory Capacity=4K2025.03 | 3.03 | |
| YARN-128k-7BContext Length=1k, Model Size=7B2026.06 | 3.03 | |
| ERMARContext Length=4k, Model Size=3B, Memory=4K2026.06 | 3.03 | |
| Phi3-128kModel Size=3B, Context Length=16k, GPU Configuration=Dual GPU2025.03 | 3.08 | |
| Phi3-128kContext Length=16k, Model Size=3B, GPU Setup=Dual2026.06 | 3.08 | |
| Phi3-128kModel Size=3B, Context Length=2k2025.03 | 3.11 | |
| Phi3-128kContext Length=2k, Model Size=3B2026.06 | 3.11 | |
| KGERMARContext Length=4k, Model Size=3B, Memory=4K2026.06 | 3.12 | |
| LongLLaMA-3B*Model Size=3B, Context Length=2k2025.03 | 3.15 | |
| MemLong (w/ 4K capacity)Model Size=3B, Context Length=4k, Memory Capacity=4K2025.03 | 3.15 | |
| LongLLaMA-3BContext Length=2k, Model Size=3B, Memory=No Extended Memory2026.06 | 3.15 | |
| MemLongContext Length=4k, Model Size=3B, Memory=4K2026.06 | 3.15 | |
| KGERMARContext Length=2k, Model Size=3B, Memory=4K2026.06 | 3.15 | |
| MemLong (w/ 4K capacity)Model Size=3B, Context Length=2k, Memory Capacity=4K2025.03 | 3.16 | |
| MemLongContext Length=2k, Model Size=3B, Memory=4K2026.06 | 3.16 | |
| KGERMARContext Length=16k, Model Size=3B, Memory=4K2026.06 | 3.17 | |
| MemLong-3B*Model Size=3B, Context Length=2k2025.03 | 3.18 | |
| ERMAR (w/ 4K capacity)Model Size=3B, Context Length=16k, Memory Capacity=4K2025.03 | 3.18 | |
| MemLong-3BContext Length=2k, Model Size=3B, Memory=No Extended Memory2026.06 | 3.18 | |
| ERMARContext Length=16k, Model Size=3B, Memory=4K2026.06 | 3.18 | |
| LongLLaMA-3B†Model Size=3B, Context Length=2k2025.03 | 3.22 | |
| LongLLaMA-3BContext Length=2k, Model Size=3B, Memory=Extended Memory2026.06 | 3.22 | |
| LongLoRA-7B-32kModel Size=7B, Context Length=4k2025.03 | 3.23 | |
| LongLoRA-7B-32kContext Length=4k, Model Size=7B2026.06 | 3.23 | |
| LLaMA-2-7BModel Size=7B, Context Length=1k2025.03 | 3.24 | |
| ERMAR (w/ 4K capacity)Model Size=3B, Context Length=1k, Memory Capacity=4K2025.03 | 3.24 | |
| LLaMA-2-7BContext Length=1k, Model Size=7B2026.06 | 3.24 | |
| ERMARContext Length=1k, Model Size=3B, Memory=4K2026.06 | 3.24 | |
| YARN-128k-7bModel Size=7B, Context Length=2k2025.03 | 3.29 | |
| YARN-128k-7BContext Length=2k, Model Size=7B2026.06 | 3.29 | |
| LongLoRA-7B-32kModel Size=7B, Context Length=2k2025.03 | 3.35 | |
| LongLoRA-7B-32kContext Length=2k, Model Size=7B2026.06 | 3.35 | |
| LLaMA-2-7BModel Size=7B, Context Length=2k2025.03 | 3.4 | |
| LLaMA-2-7BContext Length=2k, Model Size=7B2026.06 | 3.4 | |
| MemLong (w/ 4K capacity)Model Size=3B, Context Length=1k, Memory Capacity=4K2025.03 | 3.53 | |
| MemLongContext Length=1k, Model Size=3B, Memory=4K2026.06 | 3.53 | |
| LongLLaMA-3B*Model Size=3B, Context Length=1k2025.03 | 3.55 | |
| LongLLaMA-3B†Model Size=3B, Context Length=1k2025.03 | 3.55 | |
| LongLLaMA-3BContext Length=1k, Model Size=3B, Memory=No Extended Memory2026.06 | 3.55 | |
| LongLLaMA-3BContext Length=1k, Model Size=3B, Memory=Extended Memory2026.06 | 3.55 | |
| KGERMARContext Length=1k, Model Size=3B, Memory=4K2026.06 | 3.55 | |
| MemLong-3B*Model Size=3B, Context Length=1k2025.03 | 3.58 | |
| MemLong-3BContext Length=1k, Model Size=3B, Memory=No Extended Memory2026.06 | 3.58 | |
| LongLoRA-7B-32kModel Size=7B, Context Length=1k2025.03 | 3.68 | |
| LongLoRA-7B-32kContext Length=1k, Model Size=7B2026.06 | 3.68 | |
| Phi3-128kModel Size=3B, Context Length=1k2025.03 | 4.25 | |
| Phi3-128kContext Length=1k, Model Size=3B2026.06 | 4.25 | |
| PoSEContext Length=32K, Interpolation Method=Linear, Fine-tuned Length=4K2024.06 | 4.4 | |
| CREAMContext Length=32K, Interpolation Method=Linear, Fine-tuned Length=4K2024.06 | 4.4 | |
| PoSEContext Length=32K, Interpolation Method=YaRN, Fine-tuned Length=4K2024.06 | 4.4 | |
| CREAMContext Length=32K, Interpolation Method=YaRN, Fine-tuned Length=4K2024.06 | 4.4 | |
| CREAMContext Length=16K, Interpolation Method=Linear, Fine-tuned Length=4K2024.06 | 4.5 | |
| PoSEContext Length=16K, Interpolation Method=NTK, Fine-tuned Length=4K2024.06 | 4.5 | |
| CREAMContext Length=16K, Interpolation Method=NTK, Fine-tuned Length=4K2024.06 | 4.5 | |
| PoSEContext Length=16K, Interpolation Method=YaRN, Fine-tuned Length=4K2024.06 | 4.5 | |
| CREAMContext Length=16K, Interpolation Method=YaRN, Fine-tuned Length=4K2024.06 | 4.5 | |
| LLaMa2-7BContext Length=4K, Fine-tuned Length=4K2024.06 | 4.6 | |
| PoSEContext Length=8K, Interpolation Method=Linear, Fine-tuned Length=4K2024.06 | 4.6 | |
| PoSEContext Length=16K, Interpolation Method=Linear, Fine-tuned Length=4K2024.06 | 4.6 | |
| CREAMContext Length=8K, Interpolation Method=Linear, Fine-tuned Length=4K2024.06 | 4.6 |