# Language Modeling on PG-19

Source: [Language Modeling on PG-19](<https://www.sota2.com/research/sota/language-modeling-on-pg-19>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-15T07:10:52.641Z

Evaluation of language modeling performance on the PG-19 dataset, typically measured using log-perplexity (log-ppl).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Language Modeling |
| Dataset (including any stated split) | PG-19 |
| Metric | Perplexity |
| Unit | SCALAR |
| Direction | Lower is better |
| Results loaded | 295 of 295 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Perplexity](<https://www.sota2.com/research/sota/language-modeling-on-pg-19.md?metric=Perplexity>): Lower is better; unit: SCALAR.

- [Log-PPL](<https://www.sota2.com/research/sota/language-modeling-on-pg-19.md?metric=Log-PPL>): Lower is better; unit: SCALAR.

- [Log-PPL (Long)](<https://www.sota2.com/research/sota/language-modeling-on-pg-19.md?metric=Log-PPL%20(Long)>): Lower is better; unit: SCALAR.

- [PPL Change (%) vs Baseline](<https://www.sota2.com/research/sota/language-modeling-on-pg-19.md?metric=PPL%20Change%20(%25)%20vs%20Baseline>): Lower is better; unit: PERCENT.

- [Log-PPL Change](<https://www.sota2.com/research/sota/language-modeling-on-pg-19.md?metric=Log-PPL%20Change>): Lower is better; unit: SCALAR.

- [Forward BPC](<https://www.sota2.com/research/sota/language-modeling-on-pg-19.md?metric=Forward%20BPC>): Lower is better; unit: SCALAR.

- [Backward BPC](<https://www.sota2.com/research/sota/language-modeling-on-pg-19.md?metric=Backward%20BPC>): Lower is better; unit: SCALAR.

## Recorded results

| Method | Perplexity | Result date | Report paper |
| --- | --- | --- | --- |
| CREAM | 5 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.1 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.1 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.2 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.2 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.2 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.2 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.2 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.3 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.3 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.3 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.3 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.4 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.4 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.4 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.4 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.5 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 5.9 | Not recorded | 68fa751e1e209131cea65df9 |
| SHAREDLLM | 5.96 | Not recorded | 69aa5f692d603e6de3e41754 |
| CREAM | 6 | Not recorded | 68fa751e1e209131cea65df9 |
| LLaMA-3.1 | 6.03 | Not recorded | 69aa5f692d603e6de3e41754 |
| CREAM | 6.1 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 6.1 | Not recorded | 68fa751e1e209131cea65df9 |
| CEPE | 6.1 | Not recorded | 69aa5f692d603e6de3e41754 |
| SHAREDLLM | 6.28 | Not recorded | 69aa5f692d603e6de3e41754 |
| CEPE | 6.4 | Not recorded | 69aa5f692d603e6de3e41754 |
| LLaMA-3.1 | 6.52 | Not recorded | 69aa5f692d603e6de3e41754 |
| SHAREDLLM | 6.55 | Not recorded | 69aa5f692d603e6de3e41754 |
| YaRN | 6.62 | Not recorded | 69aa5f692d603e6de3e41754 |
| SHAREDLLM | 6.65 | Not recorded | 69aa5f692d603e6de3e41754 |
| CEPE | 6.69 | Not recorded | 69aa5f692d603e6de3e41754 |
| PI | 6.77 | Not recorded | 69aa5f692d603e6de3e41754 |
| LLaMA-3.1 | 6.77 | Not recorded | 69aa5f692d603e6de3e41754 |
| CEPE | 6.8 | Not recorded | 69aa5f692d603e6de3e41754 |
| LLaMA-2-32K | 6.81 | Not recorded | 69aa5f692d603e6de3e41754 |
| Full KV | 6.81 | Not recorded | 6a0fe305f984739ed6739d10 |
| LLaMA-3.1 | 6.84 | Not recorded | 69aa5f692d603e6de3e41754 |
| YaRN | 6.85 | Not recorded | 69aa5f692d603e6de3e41754 |
| PI | 6.89 | Not recorded | 69aa5f692d603e6de3e41754 |
| YaRN | 6.91 | Not recorded | 69aa5f692d603e6de3e41754 |
| LLaMA-2-32K | 6.93 | Not recorded | 69aa5f692d603e6de3e41754 |
| PI | 6.97 | Not recorded | 69aa5f692d603e6de3e41754 |
| LLaMA-2-32K | 7.04 | Not recorded | 69aa5f692d603e6de3e41754 |
| Meta-Soft | 7.05 | Not recorded | 6a0fe305f984739ed6739d10 |
| Judge Q | 7.11 | Not recorded | 6a0fe305f984739ed6739d10 |
| ZeroMerge | 7.13 | Not recorded | 6a0fe305f984739ed6739d10 |
| YARN-128k-7b | 7.17 | Not recorded | 68fa79d81e209131cea7fb95 |
| YARN-128k-7B | 7.17 | Not recorded | 6a2f8723d33dfb23e783c076 |
| YARN-128k-7b | 7.22 | Not recorded | 68fa79d81e209131cea7fb95 |
| YARN-128k-7B | 7.22 | Not recorded | 6a2f8723d33dfb23e783c076 |
| Full KV | 7.23 | Not recorded | 6a0fe305f984739ed6739d10 |
| SnapKV | 7.24 | Not recorded | 6a0fe305f984739ed6739d10 |
| H2O | 7.27 | Not recorded | 6a0fe305f984739ed6739d10 |
| YARN-128k-7b | 7.47 | Not recorded | 68fa79d81e209131cea7fb95 |
| YARN-128k-7B | 7.47 | Not recorded | 6a2f8723d33dfb23e783c076 |
| Meta-Soft | 7.49 | Not recorded | 6a0fe305f984739ed6739d10 |
| Judge Q | 7.58 | Not recorded | 6a0fe305f984739ed6739d10 |
| LLaMA-2 70B | 7.6 | Not recorded | 69d48f957d0172140ec74313 |
| LongLoRA-7B-32k | 7.62 | Not recorded | 68fa79d81e209131cea7fb95 |
| LongLoRA-7B-32k | 7.62 | Not recorded | 6a2f8723d33dfb23e783c076 |
| ZeroMerge | 7.63 | Not recorded | 6a0fe305f984739ed6739d10 |
| SnapKV | 7.72 | Not recorded | 6a0fe305f984739ed6739d10 |
| H2O | 7.79 | Not recorded | 6a0fe305f984739ed6739d10 |
| CREAM | 7.8 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 8 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 8 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 8.1 | Not recorded | 68fa751e1e209131cea65df9 |
| CREAM | 8.2 | Not recorded | 68fa751e1e209131cea65df9 |
| Focus (LLaMA-2 70B) | 8.3 | Not recorded | 69d48f957d0172140ec74313 |
| SelfExtend | 8.81 | Not recorded | 68fa78f21e209131cea7b620 |
| Dyn-NTK | 8.81 | Not recorded | 68fa78f21e209131cea7b620 |
| GALI | 8.81 | Not recorded | 68fa78f21e209131cea7b620 |
| LLaMA-2-7B | 8.92 | Not recorded | 68fa79d81e209131cea7fb95 |
| LLaMA-2-7B | 8.92 | Not recorded | 6a2f8723d33dfb23e783c076 |
| NTK | 8.95 | Not recorded | 68fa78f21e209131cea7b620 |
| YaRN | 8.97 | Not recorded | 68fa78f21e209131cea7b620 |
| SelfExtend | 8.99 | Not recorded | 68fa78f21e209131cea7b620 |
| Dyn-NTK | 8.99 | Not recorded | 68fa78f21e209131cea7b620 |
| GALI | 8.99 | Not recorded | 68fa78f21e209131cea7b620 |

### CREAM

Perplexity: 5. Result ID: 699375093a6c31484916131e.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"NTK","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"64K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5. Result ID: 699375093a6c314849161322.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"YaRN","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"64K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.1. Result ID: 699375093a6c31484916131f.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"NTK","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"96K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.1. Result ID: 699375093a6c31484916132b.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"NTK","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"64K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.2. Result ID: 699375093a6c314849161320.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"NTK","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"128K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.2. Result ID: 699375093a6c314849161321.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"NTK","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"192K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.2. Result ID: 699375093a6c314849161323.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"YaRN","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"96K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.2. Result ID: 699375093a6c314849161324.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"YaRN","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"128K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.2. Result ID: 699375093a6c314849161330.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"YaRN","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"64K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.3. Result ID: 699375093a6c314849161325.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"YaRN","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"192K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.3. Result ID: 699375093a6c31484916132c.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"NTK","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"96K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.3. Result ID: 699375093a6c31484916132d.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"NTK","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"128K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.3. Result ID: 699375093a6c314849161331.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"YaRN","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"96K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.4. Result ID: 699375093a6c31484916132e.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"NTK","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"192K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.4. Result ID: 699375093a6c31484916132f.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"NTK","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"256K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.4. Result ID: 699375093a6c314849161332.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"YaRN","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"128K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.4. Result ID: 699375093a6c314849161333.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"YaRN","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"192K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.5. Result ID: 699375093a6c314849161334.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"YaRN","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"256K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 5.9. Result ID: 699375093a6c31484916131a.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"Linear","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"64K"}. This record does not assert independent reproduction.

### SHAREDLLM

Perplexity: 5.96. Result ID: 69aa88486dbb7075b9541352.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

Recorded evaluation configuration: {"Context Length":"128K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 6. Result ID: 699375093a6c31484916131b.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"Linear","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"96K"}. This record does not assert independent reproduction.

### LLaMA-3.1

Perplexity: 6.03. Result ID: 69aa88486dbb7075b9541356.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75d11e209131cea6a028.md>)

[Implementation](<https://github.com/meta-llama/llama-agentic-system>)

Recorded evaluation configuration: {"Context Length":"128K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 6.1. Result ID: 699375093a6c31484916131c.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"Linear","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"128K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 6.1. Result ID: 699375093a6c31484916131d.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"Linear","Fine-tuning Context Length":"4K","Target Extrapolation Length":"192K","Evaluation Context Length":"192K"}. This record does not assert independent reproduction.

### CEPE

Perplexity: 6.1. Result ID: 69aa88486dbb7075b954134e.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa795c1e209131cea7d41d.md>)

Recorded evaluation configuration: {"Context Length":"128K"}. This record does not assert independent reproduction.

### SHAREDLLM

Perplexity: 6.28. Result ID: 69aa88486dbb7075b9541350.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

Recorded evaluation configuration: {"Context Length":"8K"}. This record does not assert independent reproduction.

### CEPE

Perplexity: 6.4. Result ID: 69aa88486dbb7075b954134c.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa795c1e209131cea7d41d.md>)

Recorded evaluation configuration: {"Context Length":"8K"}. This record does not assert independent reproduction.

### LLaMA-3.1

Perplexity: 6.52. Result ID: 69aa88486dbb7075b9541354.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75d11e209131cea6a028.md>)

[Implementation](<https://github.com/meta-llama/llama-agentic-system>)

Recorded evaluation configuration: {"Context Length":"8K"}. This record does not assert independent reproduction.

### SHAREDLLM

Perplexity: 6.55. Result ID: 69aa88486dbb7075b954134f.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

Recorded evaluation configuration: {"Context Length":"4K"}. This record does not assert independent reproduction.

### YaRN

Perplexity: 6.62. Result ID: 69aa88486dbb7075b9541349.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71111e209131cea4f0be.md>)

Recorded evaluation configuration: {"Context Length":"8K"}. This record does not assert independent reproduction.

### SHAREDLLM

Perplexity: 6.65. Result ID: 69aa88486dbb7075b9541351.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

Recorded evaluation configuration: {"Context Length":"32K"}. This record does not assert independent reproduction.

### CEPE

Perplexity: 6.69. Result ID: 69aa88486dbb7075b954134b.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa795c1e209131cea7d41d.md>)

Recorded evaluation configuration: {"Context Length":"4K"}. This record does not assert independent reproduction.

### PI

Perplexity: 6.77. Result ID: 69aa88486dbb7075b9541346.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6fe81e209131cea48107.md>)

Recorded evaluation configuration: {"Context Length":"8K"}. This record does not assert independent reproduction.

### LLaMA-3.1

Perplexity: 6.77. Result ID: 69aa88486dbb7075b9541353.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75d11e209131cea6a028.md>)

[Implementation](<https://github.com/meta-llama/llama-agentic-system>)

Recorded evaluation configuration: {"Context Length":"4K"}. This record does not assert independent reproduction.

### CEPE

Perplexity: 6.8. Result ID: 69aa88486dbb7075b954134d.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa795c1e209131cea7d41d.md>)

Recorded evaluation configuration: {"Context Length":"32K"}. This record does not assert independent reproduction.

### LLaMA-2-32K

Perplexity: 6.81. Result ID: 69aa88486dbb7075b9541343.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

Recorded evaluation configuration: {"Context Length":"8K"}. This record does not assert independent reproduction.

### Full KV

Perplexity: 6.81. Result ID: 6a101718fbc373592c0929e7.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

Recorded evaluation configuration: {"context length":"4k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"Full"}. This record does not assert independent reproduction.

### LLaMA-3.1

Perplexity: 6.84. Result ID: 69aa88486dbb7075b9541355.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75d11e209131cea6a028.md>)

[Implementation](<https://github.com/meta-llama/llama-agentic-system>)

Recorded evaluation configuration: {"Context Length":"32K"}. This record does not assert independent reproduction.

### YaRN

Perplexity: 6.85. Result ID: 69aa88486dbb7075b9541348.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71111e209131cea4f0be.md>)

Recorded evaluation configuration: {"Context Length":"4K"}. This record does not assert independent reproduction.

### PI

Perplexity: 6.89. Result ID: 69aa88486dbb7075b9541347.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6fe81e209131cea48107.md>)

Recorded evaluation configuration: {"Context Length":"32K"}. This record does not assert independent reproduction.

### YaRN

Perplexity: 6.91. Result ID: 69aa88486dbb7075b954134a.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71111e209131cea4f0be.md>)

Recorded evaluation configuration: {"Context Length":"32K"}. This record does not assert independent reproduction.

### LLaMA-2-32K

Perplexity: 6.93. Result ID: 69aa88486dbb7075b9541342.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

Recorded evaluation configuration: {"Context Length":"4K"}. This record does not assert independent reproduction.

### PI

Perplexity: 6.97. Result ID: 69aa88486dbb7075b9541345.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa6fe81e209131cea48107.md>)

Recorded evaluation configuration: {"Context Length":"4K"}. This record does not assert independent reproduction.

### LLaMA-2-32K

Perplexity: 7.04. Result ID: 69aa88486dbb7075b9541344.

[Reporting paper](<https://www.sota2.com/research/paper/69aa5f692d603e6de3e41754.md>)

Recorded evaluation configuration: {"Context Length":"32K"}. This record does not assert independent reproduction.

### Meta-Soft

Perplexity: 7.05. Result ID: 6a101718fbc373592c0929f1.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

Recorded evaluation configuration: {"context length":"4k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"256"}. This record does not assert independent reproduction.

### Judge Q

Perplexity: 7.11. Result ID: 6a101718fbc373592c0929ed.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7b531e209131cea87cdf.md>)

Recorded evaluation configuration: {"context length":"4k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"256"}. This record does not assert independent reproduction.

### ZeroMerge

Perplexity: 7.13. Result ID: 6a101718fbc373592c0929ef.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7be41e209131cea8b1ed.md>)

Recorded evaluation configuration: {"context length":"4k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"256"}. This record does not assert independent reproduction.

### YARN-128k-7b

Perplexity: 7.17. Result ID: 6a0c3826fbc373592c02e7f7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79d81e209131cea7fb95.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71111e209131cea4f0be.md>)

Recorded evaluation configuration: {"Model Size":"7B","Context Length":"4k","GPU Configuration":"Single GPU"}. This record does not assert independent reproduction.

### YARN-128k-7B

Perplexity: 7.17. Result ID: 6a2fa563fbc373592c2b78af.

[Reporting paper](<https://www.sota2.com/research/paper/6a2f8723d33dfb23e783c076.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71111e209131cea4f0be.md>)

Recorded evaluation configuration: {"Context Length":"4k","Model Size":"7B"}. This record does not assert independent reproduction.

### YARN-128k-7b

Perplexity: 7.22. Result ID: 6a0c3826fbc373592c02e7f5.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79d81e209131cea7fb95.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71111e209131cea4f0be.md>)

Recorded evaluation configuration: {"Model Size":"7B","Context Length":"1k","GPU Configuration":"Single GPU"}. This record does not assert independent reproduction.

### YARN-128k-7B

Perplexity: 7.22. Result ID: 6a2fa563fbc373592c2b78ad.

[Reporting paper](<https://www.sota2.com/research/paper/6a2f8723d33dfb23e783c076.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71111e209131cea4f0be.md>)

Recorded evaluation configuration: {"Context Length":"1k","Model Size":"7B"}. This record does not assert independent reproduction.

### Full KV

Perplexity: 7.23. Result ID: 6a101718fbc373592c0929e8.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

Recorded evaluation configuration: {"context length":"16k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"Full"}. This record does not assert independent reproduction.

### SnapKV

Perplexity: 7.24. Result ID: 6a101718fbc373592c0929eb.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73b21e209131cea5df89.md>)

[Implementation](<https://github.com/FasterDecoding/SnapKV>)

Recorded evaluation configuration: {"context length":"4k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"256"}. This record does not assert independent reproduction.

### H2O

Perplexity: 7.27. Result ID: 6a101718fbc373592c0929e9.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa718c1e209131cea51ead.md>)

[Implementation](<https://github.com/FMInference/H2O>)

Recorded evaluation configuration: {"context length":"4k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"256"}. This record does not assert independent reproduction.

### YARN-128k-7b

Perplexity: 7.47. Result ID: 6a0c3826fbc373592c02e7f6.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79d81e209131cea7fb95.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71111e209131cea4f0be.md>)

Recorded evaluation configuration: {"Model Size":"7B","Context Length":"2k","GPU Configuration":"Single GPU"}. This record does not assert independent reproduction.

### YARN-128k-7B

Perplexity: 7.47. Result ID: 6a2fa563fbc373592c2b78ae.

[Reporting paper](<https://www.sota2.com/research/paper/6a2f8723d33dfb23e783c076.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71111e209131cea4f0be.md>)

Recorded evaluation configuration: {"Context Length":"2k","Model Size":"7B"}. This record does not assert independent reproduction.

### Meta-Soft

Perplexity: 7.49. Result ID: 6a101718fbc373592c0929f2.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

Recorded evaluation configuration: {"context length":"16k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"256"}. This record does not assert independent reproduction.

### Judge Q

Perplexity: 7.58. Result ID: 6a101718fbc373592c0929ee.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7b531e209131cea87cdf.md>)

Recorded evaluation configuration: {"context length":"16k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"256"}. This record does not assert independent reproduction.

### LLaMA-2 70B

Perplexity: 7.6. Result ID: 69d4b3a447f34ce924bc2c74.

[Reporting paper](<https://www.sota2.com/research/paper/69d48f957d0172140ec74313.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75d11e209131cea6a028.md>)

[Implementation](<https://github.com/meta-llama/llama-agentic-system>)

Recorded evaluation configuration: {"Model":"LLaMA-2 70B"}. This record does not assert independent reproduction.

### LongLoRA-7B-32k

Perplexity: 7.62. Result ID: 6a0c3826fbc373592c02e7fb.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79d81e209131cea7fb95.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa727a1e209131cea56a34.md>)

Recorded evaluation configuration: {"Model Size":"7B","Context Length":"16k","GPU Configuration":"Single GPU"}. This record does not assert independent reproduction.

### LongLoRA-7B-32k

Perplexity: 7.62. Result ID: 6a2fa563fbc373592c2b78b3.

[Reporting paper](<https://www.sota2.com/research/paper/6a2f8723d33dfb23e783c076.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa727a1e209131cea56a34.md>)

Recorded evaluation configuration: {"Context Length":"16k","Model Size":"7B"}. This record does not assert independent reproduction.

### ZeroMerge

Perplexity: 7.63. Result ID: 6a101718fbc373592c0929f0.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7be41e209131cea8b1ed.md>)

Recorded evaluation configuration: {"context length":"16k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"256"}. This record does not assert independent reproduction.

### SnapKV

Perplexity: 7.72. Result ID: 6a101718fbc373592c0929ec.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73b21e209131cea5df89.md>)

[Implementation](<https://github.com/FasterDecoding/SnapKV>)

Recorded evaluation configuration: {"context length":"16k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"256"}. This record does not assert independent reproduction.

### H2O

Perplexity: 7.79. Result ID: 6a101718fbc373592c0929ea.

[Reporting paper](<https://www.sota2.com/research/paper/6a0fe305f984739ed6739d10.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa718c1e209131cea51ead.md>)

[Implementation](<https://github.com/FMInference/H2O>)

Recorded evaluation configuration: {"context length":"16k","Backbone":"Llama-3.1-8B-Instruct","KV cache compression (B)":"256"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 7.8. Result ID: 699375093a6c314849161326.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"Linear","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"64K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 8. Result ID: 699375093a6c314849161327.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"Linear","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"96K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 8. Result ID: 699375093a6c314849161328.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"Linear","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"128K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 8.1. Result ID: 699375093a6c314849161329.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"Linear","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"192K"}. This record does not assert independent reproduction.

### CREAM

Perplexity: 8.2. Result ID: 699375093a6c31484916132a.

[Reporting paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa751e1e209131cea65df9.md>)

Recorded evaluation configuration: {"Backbone":"LLaMa2-7B","Interpolation Method":"Linear","Fine-tuning Context Length":"4K","Target Extrapolation Length":"256K","Evaluation Context Length":"256K"}. This record does not assert independent reproduction.

### Focus (LLaMA-2 70B)

Perplexity: 8.3. Result ID: 69d4b3a447f34ce924bc2c75.

[Reporting paper](<https://www.sota2.com/research/paper/69d48f957d0172140ec74313.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69d48f957d0172140ec74313.md>)

Recorded evaluation configuration: {"Backbone":"LLaMA-2 70B","Centroid params":"10.5M (0.015%)","K":"4","dg":"16","top-k":"2"}. This record does not assert independent reproduction.

### SelfExtend

Perplexity: 8.81. Result ID: 6995b2a23badfaffe6b2830d.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b620.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74131e209131cea5fa75.md>)

Recorded evaluation configuration: {"Backbone":"Llama2-7b-chat-4k","Context Window":"1k"}. This record does not assert independent reproduction.

### Dyn-NTK

Perplexity: 8.81. Result ID: 6995b2a23badfaffe6b28325.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b620.md>)

Recorded evaluation configuration: {"Backbone":"Llama2-7b-chat-4k","Context Window":"1k"}. This record does not assert independent reproduction.

### GALI

Perplexity: 8.81. Result ID: 6995b2a23badfaffe6b28335.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b620.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b620.md>)

[Implementation](<https://github.com/adlnlp/Gali>)

Recorded evaluation configuration: {"Backbone":"Llama2-7b-chat-4k","Context Window":"1k"}. This record does not assert independent reproduction.

### LLaMA-2-7B

Perplexity: 8.92. Result ID: 6a0c3826fbc373592c02e7fe.

[Reporting paper](<https://www.sota2.com/research/paper/68fa79d81e209131cea7fb95.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa701f1e209131cea492fe.md>)

[Implementation](<https://github.com/facebookresearch/llama>)

Recorded evaluation configuration: {"Model Size":"7B","Context Length":"4k","GPU Configuration":"Single GPU"}. This record does not assert independent reproduction.

### LLaMA-2-7B

Perplexity: 8.92. Result ID: 6a2fa563fbc373592c2b78b6.

[Reporting paper](<https://www.sota2.com/research/paper/6a2f8723d33dfb23e783c076.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa701f1e209131cea492fe.md>)

[Implementation](<https://github.com/facebookresearch/llama>)

Recorded evaluation configuration: {"Context Length":"4k","Model Size":"7B"}. This record does not assert independent reproduction.

### NTK

Perplexity: 8.95. Result ID: 6995b2a23badfaffe6b2831d.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b620.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74131e209131cea5fa75.md>)

Recorded evaluation configuration: {"Backbone":"Llama2-7b-chat-4k","Context Window":"1k"}. This record does not assert independent reproduction.

### YaRN

Perplexity: 8.97. Result ID: 6995b2a23badfaffe6b2832e.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b620.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa71111e209131cea4f0be.md>)

Recorded evaluation configuration: {"Backbone":"Llama2-7b-chat-4k","Context Window":"2k"}. This record does not assert independent reproduction.

### SelfExtend

Perplexity: 8.99. Result ID: 6995b2a23badfaffe6b2830e.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b620.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74131e209131cea5fa75.md>)

Recorded evaluation configuration: {"Backbone":"Llama2-7b-chat-4k","Context Window":"2k"}. This record does not assert independent reproduction.

### Dyn-NTK

Perplexity: 8.99. Result ID: 6995b2a23badfaffe6b28326.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b620.md>)

Recorded evaluation configuration: {"Backbone":"Llama2-7b-chat-4k","Context Window":"2k"}. This record does not assert independent reproduction.

### GALI

Perplexity: 8.99. Result ID: 6995b2a23badfaffe6b28336.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b620.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b620.md>)

[Implementation](<https://github.com/adlnlp/Gali>)

Recorded evaluation configuration: {"Backbone":"Llama2-7b-chat-4k","Context Window":"2k"}. This record does not assert independent reproduction.

Page 1 of 4. This is an index page, not the full collection.

- [Next index page](<https://www.sota2.com/research/sota/language-modeling-on-pg-19.md?metric=Perplexity&page=2>): Follow this link to discover more pages.
