# Long-context language modeling on RULER 1.0 (test)

Source: [Long-context language modeling on RULER 1.0 (test)](<https://www.sota2.com/research/sota/long-context-language-modeling-on-ruler-1-0-test>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:34:02.050Z

Evaluation of model performance on the RULER 1.0 test set, specifically measuring accuracy across various long context window sizes (4K up to 128K tokens).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Long-context language modeling |
| Dataset (including any stated split) | RULER 1.0 (test) |
| Metric | Accuracy (4K Context) |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 16 of 16 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Accuracy (4K Context)](<https://www.sota2.com/research/sota/long-context-language-modeling-on-ruler-1-0-test.md?metric=Accuracy%20(4K%20Context)>): Higher is better; unit: PERCENT.

- [Accuracy (8K Context)](<https://www.sota2.com/research/sota/long-context-language-modeling-on-ruler-1-0-test.md?metric=Accuracy%20(8K%20Context)>): Higher is better; unit: PERCENT.

- [Accuracy (16K Context)](<https://www.sota2.com/research/sota/long-context-language-modeling-on-ruler-1-0-test.md?metric=Accuracy%20(16K%20Context)>): Higher is better; unit: PERCENT.

- [Accuracy (32K Context)](<https://www.sota2.com/research/sota/long-context-language-modeling-on-ruler-1-0-test.md?metric=Accuracy%20(32K%20Context)>): Higher is better; unit: PERCENT.

- [Accuracy (64K Context)](<https://www.sota2.com/research/sota/long-context-language-modeling-on-ruler-1-0-test.md?metric=Accuracy%20(64K%20Context)>): Higher is better; unit: PERCENT.

- [Accuracy (128K Context)](<https://www.sota2.com/research/sota/long-context-language-modeling-on-ruler-1-0-test.md?metric=Accuracy%20(128K%20Context)>): Higher is better; unit: PERCENT.

- [Average Accuracy](<https://www.sota2.com/research/sota/long-context-language-modeling-on-ruler-1-0-test.md?metric=Average%20Accuracy>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Accuracy (4K Context) | Result date | Report paper |
| --- | --- | --- | --- |
| MInference | 0.977 | Not recorded | 68fa75761e209131cea681f8 |
| LLAMA-3-8B-262K | 0.972 | Not recorded | 68fa75761e209131cea681f8 |
| StreamingLLM | 0.972 | Not recorded | 68fa75761e209131cea681f8 |
| InfLLM | 0.947 | Not recorded | 68fa75761e209131cea681f8 |
| MInference | 0.946 | Not recorded | 68fa75761e209131cea681f8 |
| GLM-4-9B-1M | 0.938 | Not recorded | 68fa75761e209131cea681f8 |
| StreamingLLM | 0.938 | Not recorded | 68fa75761e209131cea681f8 |
| MInference | 0.923 | Not recorded | 68fa75761e209131cea681f8 |
| Yi-9B-200K | 0.919 | Not recorded | 68fa75761e209131cea681f8 |
| StreamingLLM | 0.919 | Not recorded | 68fa75761e209131cea681f8 |
| InfLLM | 0.894 | Not recorded | 68fa75761e209131cea681f8 |
| InfLLM | 0.803 | Not recorded | 68fa75761e209131cea681f8 |
| StreamingLLM w/ dilated | 0.448 | Not recorded | 68fa75761e209131cea681f8 |
| StreamingLLM w/ dilated | 0.234 | Not recorded | 68fa75761e209131cea681f8 |
| StreamingLLM w/ strided | 0.026 | Not recorded | 68fa75761e209131cea681f8 |
| StreamingLLM w/ strided | 0.02 | Not recorded | 68fa75761e209131cea681f8 |

### MInference

Accuracy (4K Context): 0.977. Result ID: 6993c7223a6c314849187742.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Implementation](<https://aka.ms/MInference>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"32K","Backbone":"LLAMA-3-8B-262K"}. This record does not assert independent reproduction.

### LLAMA-3-8B-262K

Accuracy (4K Context): 0.972. Result ID: 6993c7223a6c31484918773d.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

Recorded evaluation configuration: {"Claimed context length":"262K","Effective context length":"16K","Backbone":"LLAMA-3-8B-262K"}. This record does not assert independent reproduction.

### StreamingLLM

Accuracy (4K Context): 0.972. Result ID: 6993c7223a6c31484918773e.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72d71e209131cea58d79.md>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"4K","Backbone":"LLAMA-3-8B-262K"}. This record does not assert independent reproduction.

### InfLLM

Accuracy (4K Context): 0.947. Result ID: 6993c7223a6c31484918774b.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa736c1e209131cea5c6bf.md>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"8K","Backbone":"GLM-4-9B-1M"}. This record does not assert independent reproduction.

### MInference

Accuracy (4K Context): 0.946. Result ID: 6993c7223a6c31484918774c.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Implementation](<https://aka.ms/MInference>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"64K","Backbone":"GLM-4-9B-1M"}. This record does not assert independent reproduction.

### GLM-4-9B-1M

Accuracy (4K Context): 0.938. Result ID: 6993c7223a6c314849187749.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa74521e209131cea610b6.md>)

Recorded evaluation configuration: {"Claimed context length":"1M","Effective context length":"64K","Backbone":"GLM-4-9B-1M"}. This record does not assert independent reproduction.

### StreamingLLM

Accuracy (4K Context): 0.938. Result ID: 6993c7223a6c31484918774a.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72d71e209131cea58d79.md>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"4K","Backbone":"GLM-4-9B-1M"}. This record does not assert independent reproduction.

### MInference

Accuracy (4K Context): 0.923. Result ID: 6993c7223a6c314849187748.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Implementation](<https://aka.ms/MInference>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"8K","Backbone":"Yi-9B-200K"}. This record does not assert independent reproduction.

### Yi-9B-200K

Accuracy (4K Context): 0.919. Result ID: 6993c7223a6c314849187743.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa76a71e209131cea6e366.md>)

[Implementation](<https://github.com/01-ai/Yi>)

Recorded evaluation configuration: {"Claimed context length":"200K","Effective context length":"8K","Backbone":"Yi-9B-200K"}. This record does not assert independent reproduction.

### StreamingLLM

Accuracy (4K Context): 0.919. Result ID: 6993c7223a6c314849187744.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa72d71e209131cea58d79.md>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"4K","Backbone":"Yi-9B-200K"}. This record does not assert independent reproduction.

### InfLLM

Accuracy (4K Context): 0.894. Result ID: 6993c7223a6c314849187741.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa736c1e209131cea5c6bf.md>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"4K","Backbone":"LLAMA-3-8B-262K"}. This record does not assert independent reproduction.

### InfLLM

Accuracy (4K Context): 0.803. Result ID: 6993c7223a6c314849187747.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa736c1e209131cea5c6bf.md>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"\<4K","Backbone":"Yi-9B-200K"}. This record does not assert independent reproduction.

### StreamingLLM w/ dilated

Accuracy (4K Context): 0.448. Result ID: 6993c7223a6c314849187745.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa68d41e209131cea1e701.md>)

[Implementation](<https://github.com/allenai/longformer>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"\<4K","Backbone":"Yi-9B-200K"}. This record does not assert independent reproduction.

### StreamingLLM w/ dilated

Accuracy (4K Context): 0.234. Result ID: 6993c7223a6c31484918773f.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa68d41e209131cea1e701.md>)

[Implementation](<https://github.com/allenai/longformer>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"\<4K","Backbone":"LLAMA-3-8B-262K"}. This record does not assert independent reproduction.

### StreamingLLM w/ strided

Accuracy (4K Context): 0.026. Result ID: 6993c7223a6c314849187746.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65aa1e209131cea0bc26.md>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"\<4K","Backbone":"Yi-9B-200K"}. This record does not assert independent reproduction.

### StreamingLLM w/ strided

Accuracy (4K Context): 0.02. Result ID: 6993c7223a6c314849187740.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa65aa1e209131cea0bc26.md>)

Recorded evaluation configuration: {"Claimed context length":"-","Effective context length":"\<4K","Backbone":"LLAMA-3-8B-262K"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
