Language Modeling on PG-19
5PerplexityCREAM
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=64K2024.06 | 5 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=64K2024.06 | 5 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=96K2024.06 | 5.1 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=64K2024.06 | 5.1 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=128K2024.06 | 5.2 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=192K2024.06 | 5.2 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=96K2024.06 | 5.2 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=128K2024.06 | 5.2 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=64K2024.06 | 5.2 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=192K2024.06 | 5.3 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=96K2024.06 | 5.3 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=128K2024.06 | 5.3 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=96K2024.06 | 5.3 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=192K2024.06 | 5.4 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=NTK, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=256K2024.06 | 5.4 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=128K2024.06 | 5.4 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=192K2024.06 | 5.4 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=YaRN, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=256K2024.06 | 5.5 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=64K2024.06 | 5.9 | — | — | — | — | — | — | |
| SHAREDLLMContext Length=128K2026.03 | 5.96 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=96K2024.06 | 6 | — | — | — | — | — | — | |
| LLaMA-3.1Context Length=128K2026.03 | 6.03 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=128K2024.06 | 6.1 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=192K, Evaluation Context Length=192K2024.06 | 6.1 | — | — | — | — | — | — | |
| CEPEContext Length=128K2026.03 | 6.1 | — | — | — | — | — | — | |
| SHAREDLLMContext Length=8K2026.03 | 6.28 | — | — | — | — | — | — | |
| CEPEContext Length=8K2026.03 | 6.4 | — | — | — | — | — | — | |
| LLaMA-3.1Context Length=8K2026.03 | 6.52 | — | — | — | — | — | — | |
| SHAREDLLMContext Length=4K2026.03 | 6.55 | — | — | — | — | — | — | |
| YaRNContext Length=8K2026.03 | 6.62 | — | — | — | — | — | — | |
| SHAREDLLMContext Length=32K2026.03 | 6.65 | — | — | — | — | — | — | |
| CEPEContext Length=4K2026.03 | 6.69 | — | — | — | — | — | — | |
| PIContext Length=8K2026.03 | 6.77 | — | — | — | — | — | — | |
| LLaMA-3.1Context Length=4K2026.03 | 6.77 | — | — | — | — | — | — | |
| CEPEContext Length=32K2026.03 | 6.8 | — | — | — | — | — | — | |
| LLaMA-2-32KContext Length=8K2026.03 | 6.81 | — | — | — | — | — | — | |
| Full KVcontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=Full2026.05 | 6.81 | — | — | — | — | — | — | |
| LLaMA-3.1Context Length=32K2026.03 | 6.84 | — | — | — | — | — | — | |
| YaRNContext Length=4K2026.03 | 6.85 | — | — | — | — | — | — | |
| PIContext Length=32K2026.03 | 6.89 | — | — | — | — | — | — | |
| YaRNContext Length=32K2026.03 | 6.91 | — | — | — | — | — | — | |
| LLaMA-2-32KContext Length=4K2026.03 | 6.93 | — | — | — | — | — | — | |
| PIContext Length=4K2026.03 | 6.97 | — | — | — | — | — | — | |
| LLaMA-2-32KContext Length=32K2026.03 | 7.04 | — | — | — | — | — | — | |
| Meta-Softcontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 7.05 | — | — | — | — | — | — | |
| Judge Qcontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 7.11 | — | — | — | — | — | — | |
| ZeroMergecontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 7.13 | — | — | — | — | — | — | |
| YARN-128k-7bModel Size=7B, Context Length=4k, GPU Configuration=Single GPU2025.03 | 7.17 | — | — | — | — | — | — | |
| YARN-128k-7BContext Length=4k, Model Size=7B2026.06 | 7.17 | — | — | — | — | — | — | |
| YARN-128k-7bModel Size=7B, Context Length=1k, GPU Configuration=Single GPU2025.03 | 7.22 | — | — | — | — | — | — | |
| YARN-128k-7BContext Length=1k, Model Size=7B2026.06 | 7.22 | — | — | — | — | — | — | |
| Full KVcontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=Full2026.05 | 7.23 | — | — | — | — | — | — | |
| SnapKVcontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 7.24 | — | — | — | — | — | — | |
| H2Ocontext length=4k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 7.27 | — | — | — | — | — | — | |
| YARN-128k-7bModel Size=7B, Context Length=2k, GPU Configuration=Single GPU2025.03 | 7.47 | — | — | — | — | — | — | |
| YARN-128k-7BContext Length=2k, Model Size=7B2026.06 | 7.47 | — | — | — | — | — | — | |
| Meta-Softcontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 7.49 | — | — | — | — | — | — | |
| Judge Qcontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 7.58 | — | — | — | — | — | — | |
| LLaMA-2 70BModel=LLaMA-2 70B2026.03 | 7.6 | — | — | — | — | — | — | |
| LongLoRA-7B-32kModel Size=7B, Context Length=16k, GPU Configuration=Single GPU2025.03 | 7.62 | — | — | — | — | — | — | |
| LongLoRA-7B-32kContext Length=16k, Model Size=7B2026.06 | 7.62 | — | — | — | — | — | — | |
| ZeroMergecontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 7.63 | — | — | — | — | — | — | |
| SnapKVcontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 7.72 | — | — | — | — | — | — | |
| H2Ocontext length=16k, Backbone=Llama-3.1-8B-Instruct, KV cache compression (B)=2562026.05 | 7.79 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=64K2024.06 | 7.8 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=96K2024.06 | 8 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=128K2024.06 | 8 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=192K2024.06 | 8.1 | — | — | — | — | — | — | |
| CREAMBackbone=LLaMa2-7B, Interpolation Method=Linear, Fine-tuning Context Length=4K, Target Extrapolation Length=256K, Evaluation Context Length=256K2024.06 | 8.2 | — | — | — | — | — | — | |
| Focus (LLaMA-2 70B)Backbone=LLaMA-2 70B, Centroid params=10.5M (0.015%), K=4, dg=16, top-k=22026.03 | 8.3 | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama2-7b-chat-4k, Context Window=1k2025.02 | 8.81 | — | — | — | — | — | — | |
| Dyn-NTKBackbone=Llama2-7b-chat-4k, Context Window=1k2025.02 | 8.81 | — | — | — | — | — | — | |
| GALIBackbone=Llama2-7b-chat-4k, Context Window=1k2025.02 | 8.81 | — | — | — | — | — | — | |
| LLaMA-2-7BModel Size=7B, Context Length=4k, GPU Configuration=Single GPU2025.03 | 8.92 | — | — | — | — | — | — | |
| LLaMA-2-7BContext Length=4k, Model Size=7B2026.06 | 8.92 | — | — | — | — | — | — | |
| NTKBackbone=Llama2-7b-chat-4k, Context Window=1k2025.02 | 8.95 | — | — | — | — | — | — | |
| YaRNBackbone=Llama2-7b-chat-4k, Context Window=2k2025.02 | 8.97 | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama2-7b-chat-4k, Context Window=2k2025.02 | 8.99 | — | — | — | — | — | — | |
| Dyn-NTKBackbone=Llama2-7b-chat-4k, Context Window=2k2025.02 | 8.99 | — | — | — | — | — | — | |
| GALIBackbone=Llama2-7b-chat-4k, Context Window=2k2025.02 | 8.99 | — | — | — | — | — | — | |
| YaRNBackbone=Llama2-7b-chat-4k, Context Window=7k2025.02 | 9.03 | — | — | — | — | — | — | |
| NTKBackbone=Llama2-7b-chat-4k, Context Window=2k2025.02 | 9.04 | — | — | — | — | — | — | |
| YaRNBackbone=Llama2-7b-chat-4k, Context Window=8k2025.02 | 9.04 | — | — | — | — | — | — | |
| NTKBackbone=Llama2-7b-chat-4k, Context Window=6k2025.02 | 9.06 | — | — | — | — | — | — | |
| ChunkLlamaBackbone=Llama2-7b-chat-4k, Context Window=1k2025.02 | 9.07 | — | — | — | — | — | — | |
| YaRNBackbone=Llama2-7b-chat-4k, Context Window=3k2025.02 | 9.07 | — | — | — | — | — | — | |
| Dyn-NTKBackbone=Llama2-7b-chat-4k, Context Window=3k2025.02 | 9.15 | — | — | — | — | — | — | |
| YaRNBackbone=Llama2-7b-chat-4k, Context Window=6k2025.02 | 9.15 | — | — | — | — | — | — | |
| GALIBackbone=Llama2-7b-chat-4k, Context Window=3k2025.02 | 9.15 | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama2-7b-chat-4k, Context Window=3k2025.02 | 9.16 | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama2-7b-chat-4k, Context Window=6k2025.02 | 9.16 | — | — | — | — | — | — | |
| NTKBackbone=Llama2-7b-chat-4k, Context Window=3k2025.02 | 9.16 | — | — | — | — | — | — | |
| NTKBackbone=Llama2-7b-chat-4k, Context Window=5k2025.02 | 9.16 | — | — | — | — | — | — | |
| YaRNBackbone=Llama2-7b-chat-4k, Context Window=4k2025.02 | 9.16 | — | — | — | — | — | — | |
| YaRNBackbone=Llama2-7b-chat-4k, Context Window=5k2025.02 | 9.17 | — | — | — | — | — | — | |
| NTKBackbone=Llama2-7b-chat-4k, Context Window=4k2025.02 | 9.18 | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama2-7b-chat-4k, Context Window=7k2025.02 | 9.2 | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama2-7b-chat-4k, Context Window=4k2025.02 | 9.24 | — | — | — | — | — | — | |
| GALIBackbone=Llama2-7b-chat-4k, Context Window=4k2025.02 | 9.24 | — | — | — | — | — | — | |
| SelfExtendBackbone=Llama2-7b-chat-4k, Context Window=5k2025.02 | 9.25 | — | — | — | — | — | — |