Language Modeling on WikiText-103 (PPL and FLOPs)
5.47Perplexity (PPL)LongLoRA-7B-32k
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LongLoRA-7B-32kModel Size=7B, Context Length=16k2025.03 | 5.47 | — | |
| LLaMA-2-7BModel Size=7B, Context Length=4k2025.03 | 5.66 | — | |
| YARN-128k-7bModel Size=7B, Context Length=1k2025.03 | 5.71 | — | |
| YARN-128k-7bModel Size=7B, Context Length=4k2025.03 | 5.71 | — | |
| YARN-128k-7bModel Size=7B, Context Length=2k2025.03 | 6.11 | — | |
| LLaMA-2-7BModel Size=7B, Context Length=2k2025.03 | 6.49 | — | |
| Phi3-128kModel Size=3B, Context Length=4k2025.03 | 7.01 | — | |
| Phi3-128kModel Size=3B, Context Length=16k, GPU Configuration=Dual GPU2025.03 | 7.2 | — | |
| Phi3-128kModel Size=3B, Context Length=2k2025.03 | 7.22 | — | |
| Phi3-128kModel Size=3B, Context Length=1k2025.03 | 7.54 | — | |
| ERMAR (w/ 4K capacity)Model Size=3B, Context Length=2k, Memory Capacity=4K2025.03 | 7.61 | — | |
| ERMAR (w/ 4K capacity)Model Size=3B, Context Length=4k, Memory Capacity=4K2025.03 | 7.62 | — | |
| ERMAR (w/ 4K capacity)Model Size=3B, Context Length=16k, Memory Capacity=4K2025.03 | 7.8 | — | |
| LongLoRA-7B-32kModel Size=7B, Context Length=2k2025.03 | 7.83 | — | |
| LongLLaMA-3B†Model Size=3B, Context Length=4k2025.03 | 7.84 | — | |
| MemLong (w/ 4K capacity)Model Size=3B, Context Length=4k, Memory Capacity=4K2025.03 | 7.87 | — | |
| MemLong (w/ 4K capacity)Model Size=3B, Context Length=2k, Memory Capacity=4K2025.03 | 7.92 | — | |
| MemLong-3B*Model Size=3B, Context Length=2k2025.03 | 7.93 | — | |
| LongLoRA-7B-32kModel Size=7B, Context Length=1k2025.03 | 7.99 | — | |
| MemLong (w/ 4K capacity)Model Size=3B, Context Length=16k, Memory Capacity=4K2025.03 | 7.99 | — | |
| LongLLaMA-3B*Model Size=3B, Context Length=2k2025.03 | 8.07 | — | |
| OpenLLaMA-3BModel Size=3B, Context Length=2k2025.03 | 8.08 | — | |
| LongLLaMA-3B†Model Size=3B, Context Length=2k2025.03 | 8.33 | — | |
| LongLoRA-7B-32kModel Size=7B, Context Length=4k2025.03 | 8.39 | — | |
| ERMAR (w/ 4K capacity)Model Size=3B, Context Length=1k, Memory Capacity=4K2025.03 | 8.42 | — | |
| MemLong (w/ 4K capacity)Model Size=3B, Context Length=1k, Memory Capacity=4K2025.03 | 8.53 | — | |
| MemLong-3B*Model Size=3B, Context Length=1k2025.03 | 8.72 | — | |
| LongLLaMA-3B*Model Size=3B, Context Length=1k2025.03 | 8.88 | — | |
| OpenLLaMA-3BModel Size=3B, Context Length=4k2025.03 | 10 | — | |
| LongLLaMA-3B*Model Size=3B, Context Length=4k2025.03 | 10 | — | |
| MemLong-3B*Model Size=3B, Context Length=4k2025.03 | 10 | — | |
| OpenLLaMA-3BModel Size=3B, Context Length=1k2025.03 | 10.57 | — | |
| LongLLaMA-3B†Model Size=3B, Context Length=1k2025.03 | 10.69 | — | |
| LLaMA-2-7BModel Size=7B, Context Length=1k2025.03 | 10.82 | — | |
| AIR-MoESize=Medium2026.05 | 18.62 | 755.9 | |
| PEERSize=Medium2026.05 | 18.71 | 751 | |
| Coarse MoESize=Medium2026.05 | 18.79 | 836.7 | |
| Hierarchical MoESize=Medium2026.05 | 19.27 | 954.7 | |
| Standard Granular MoESize=Small2026.05 | 21.34 | 467.5 | |
| AIR-MoESize=Small2026.05 | 21.82 | 324.1 | |
| PEERSize=Small2026.05 | 22.25 | 352.8 | |
| Hierarchical MoESize=Small2026.05 | 22.55 | 350.4 | |
| Coarse MoESize=Small2026.05 | 23.84 | 342.5 |