# Long-context language modeling and retrieval on RULER

Source: [Long-context language modeling and retrieval on RULER](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T06:08:05.236Z

Evaluation of model performance on the RULER dataset, focusing on long-context language modeling and retrieval capabilities, measured across detailed sub-metrics including SG, MK, and QA scores.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Long-context language modeling and retrieval |
| Dataset (including any stated split) | RULER |
| Metric | VT Score |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 14 of 14 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [VT Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=VT%20Score>): Higher is better; unit: PERCENT.

- [SG1 Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=SG1%20Score>): Higher is better; unit: PERCENT.

- [SG2 Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=SG2%20Score>): Higher is better; unit: PERCENT.

- [SG3 Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=SG3%20Score>): Higher is better; unit: PERCENT.

- [MK1 Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=MK1%20Score>): Higher is better; unit: PERCENT.

- [MK2 Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=MK2%20Score>): Higher is better; unit: PERCENT.

- [MK3 Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=MK3%20Score>): Higher is better; unit: PERCENT.

- [MV Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=MV%20Score>): Higher is better; unit: PERCENT.

- [MQ Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=MQ%20Score>): Higher is better; unit: PERCENT.

- [CWE Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=CWE%20Score>): Higher is better; unit: PERCENT.

- [FWE Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=FWE%20Score>): Higher is better; unit: PERCENT.

- [QA1 Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=QA1%20Score>): Higher is better; unit: PERCENT.

- [QA2 Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=QA2%20Score>): Higher is better; unit: PERCENT.

- [Average Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=Average%20Score>): Higher is better; unit: PERCENT.

- [Single Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=Single%20Score>): Higher is better; unit: PERCENT.

- [Multi-Key Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=Multi-Key%20Score>): Higher is better; unit: PERCENT.

- [Multi-Value Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=Multi-Value%20Score>): Higher is better; unit: PERCENT.

- [Multi-Query Score](<https://www.sota2.com/research/sota/long-context-language-modeling-and-retrieval-on-ruler.md?metric=Multi-Query%20Score>): Higher is better; unit: PERCENT.

## Recorded results

| Method | VT Score | Result date | Report paper |
| --- | --- | --- | --- |
| APB | 96.4 | Not recorded | 68fa78c51e209131cea7a69e |
| FULLATTN | 89.52 | Not recorded | 68fa78c51e209131cea7a69e |
| MINFERENCE | 89.44 | Not recorded | 68fa78c51e209131cea7a69e |
| FULLATTN | 85.56 | Not recorded | 68fa78c51e209131cea7a69e |
| STARATTN | 83.96 | Not recorded | 68fa78c51e209131cea7a69e |
| STARATTN | 82.44 | Not recorded | 68fa78c51e209131cea7a69e |
| MINFERENCE | 73.68 | Not recorded | 68fa78c51e209131cea7a69e |
| MINFERENCE | 66.4 | Not recorded | 68fa78c51e209131cea7a69e |
| FULLATTN | 60.98 | Not recorded | 68fa78c51e209131cea7a69e |
| APB | 60.48 | Not recorded | 68fa78c51e209131cea7a69e |
| STARATTN | 59.4 | Not recorded | 68fa78c51e209131cea7a69e |
| APB | 51.96 | Not recorded | 68fa78c51e209131cea7a69e |
| Full Attention | 19.4 | Not recorded | 69842426a8b01e76cc8ee7de |
| LycheeDecode | 19.4 | Not recorded | 69842426a8b01e76cc8ee7de |

### APB

VT Score: 96.4. Result ID: 6995a24a3badfaffe69f32ad.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Implementation](<https://github.com/thunlp/APB>)

Recorded evaluation configuration: {"Backbone":"Qwen-2.5-14B-instruct"}. This record does not assert independent reproduction.

### FULLATTN

VT Score: 89.52. Result ID: 6995a24a3badfaffe69f32aa.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

Recorded evaluation configuration: {"Backbone":"Qwen-2.5-14B-instruct"}. This record does not assert independent reproduction.

### MINFERENCE

VT Score: 89.44. Result ID: 6995a24a3badfaffe69f32ab.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Implementation](<https://aka.ms/MInference>)

Recorded evaluation configuration: {"Backbone":"Qwen-2.5-14B-instruct"}. This record does not assert independent reproduction.

### FULLATTN

VT Score: 85.56. Result ID: 6995a24a3badfaffe69f32ae.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

Recorded evaluation configuration: {"Backbone":"Yi-34B-200K"}. This record does not assert independent reproduction.

### STARATTN

VT Score: 83.96. Result ID: 6995a24a3badfaffe69f32b0.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b606.md>)

Recorded evaluation configuration: {"Backbone":"Yi-34B-200K"}. This record does not assert independent reproduction.

### STARATTN

VT Score: 82.44. Result ID: 6995a24a3badfaffe69f32ac.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b606.md>)

Recorded evaluation configuration: {"Backbone":"Qwen-2.5-14B-instruct"}. This record does not assert independent reproduction.

### MINFERENCE

VT Score: 73.68. Result ID: 6995a24a3badfaffe69f32af.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Implementation](<https://aka.ms/MInference>)

Recorded evaluation configuration: {"Backbone":"Yi-34B-200K"}. This record does not assert independent reproduction.

### MINFERENCE

VT Score: 66.4. Result ID: 6995a24a3badfaffe69f32a7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75761e209131cea681f8.md>)

[Implementation](<https://aka.ms/MInference>)

Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-instruct"}. This record does not assert independent reproduction.

### FULLATTN

VT Score: 60.98. Result ID: 6995a24a3badfaffe69f32a6.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-instruct"}. This record does not assert independent reproduction.

### APB

VT Score: 60.48. Result ID: 6995a24a3badfaffe69f32b1.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Implementation](<https://github.com/thunlp/APB>)

Recorded evaluation configuration: {"Backbone":"Yi-34B-200K"}. This record does not assert independent reproduction.

### STARATTN

VT Score: 59.4. Result ID: 6995a24a3badfaffe69f32a8.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78f21e209131cea7b606.md>)

Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-instruct"}. This record does not assert independent reproduction.

### APB

VT Score: 51.96. Result ID: 6995a24a3badfaffe69f32a9.

[Reporting paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa78c51e209131cea7a69e.md>)

[Implementation](<https://github.com/thunlp/APB>)

Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-instruct"}. This record does not assert independent reproduction.

### Full Attention

VT Score: 19.4. Result ID: 69964c7a5698665ec490e6b1.

[Reporting paper](<https://www.sota2.com/research/paper/69842426a8b01e76cc8ee7de.md>)

Recorded evaluation configuration: {"Attention mechanism":"Full Attention","Budget":"Full context","Backbone":"Llama3-8B-Instruct-Gradient-1048k"}. This record does not assert independent reproduction.

### LycheeDecode

VT Score: 19.4. Result ID: 69964c7a5698665ec490e6b2.

[Reporting paper](<https://www.sota2.com/research/paper/69842426a8b01e76cc8ee7de.md>)

[Method origin paper](<https://www.sota2.com/research/paper/69842426a8b01e76cc8ee7de.md>)

Recorded evaluation configuration: {"Attention mechanism":"Sparse Attention","Budget":"4096 tokens","Backbone":"Llama3-8B-Instruct-Gradient-1048k"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.