# Long-context Language Modeling on RULER (4K Context) Source: [Long-context Language Modeling on RULER (4K Context)]() Collection index: [llms.txt]() Data updated: 2026-06-01T07:14:12.916Z Evaluation of model performance on the RULER dataset specifically using a 4K context window length for Long-context Language Modeling. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Long-context Language Modeling | | Dataset (including any stated split) | RULER 4K | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 29 of 29 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | Full KV | 95.3 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-V&K | 82 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-K | 81.3 | Not recorded | 68fa7c051e209131cea8be9b | | TOVA + OBCACHE-V&K | 76.7 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-V | 76 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-V&K | 75.6 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-K | 75.4 | Not recorded | 68fa7c051e209131cea8be9b | | H2O | 75.3 | Not recorded | 68fa7c051e209131cea8be9b | | TOVA | 74.5 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-V | 68.6 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-V&K | 67.8 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-K | 67.6 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-V&K | 67.5 | Not recorded | 68fa7c051e209131cea8be9b | | H2O | 67.1 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-K | 65.9 | Not recorded | 68fa7c051e209131cea8be9b | | TOVA + OBCACHE-V&K | 65.5 | Not recorded | 68fa7c051e209131cea8be9b | | TOVA | 61.8 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-V | 59.9 | Not recorded | 68fa7c051e209131cea8be9b | | H2O | 57.5 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-V | 57.1 | Not recorded | 68fa7c051e209131cea8be9b | | AdaKV + OBCACHE-V&K | 55.2 | Not recorded | 68fa7c051e209131cea8be9b | | H2O | 54.2 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-K | 47.7 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-V&K | 46.3 | Not recorded | 68fa7c051e209131cea8be9b | | H2O + OBCACHE-V | 37.7 | Not recorded | 68fa7c051e209131cea8be9b | | H2O | 33.5 | Not recorded | 68fa7c051e209131cea8be9b | | SnapKV + OBCACHE-K | 24.1 | Not recorded | 68fa7c051e209131cea8be9b | | AdaKV | 21.5 | Not recorded | 68fa7c051e209131cea8be9b | | SnapKV | 19.4 | Not recorded | 68fa7c051e209131cea8be9b | ### Full KV Accuracy: 95.3. Result ID: 6a1d3110fbc373592c15eacc. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","KV Budget (%)":"100"}. This record does not assert independent reproduction. ### H2O + OBCACHE-V&K Accuracy: 82. Result ID: 6a1d3110fbc373592c15eadf. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"40"}. This record does not assert independent reproduction. ### H2O + OBCACHE-K Accuracy: 81.3. Result ID: 6a1d3110fbc373592c15eada. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"40"}. This record does not assert independent reproduction. ### TOVA + OBCACHE-V&K Accuracy: 76.7. Result ID: 6a1d3110fbc373592c15eae4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"Avg."}. This record does not assert independent reproduction. ### H2O + OBCACHE-V Accuracy: 76. Result ID: 6a1d3110fbc373592c15ead5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"40"}. This record does not assert independent reproduction. ### H2O + OBCACHE-V&K Accuracy: 75.6. Result ID: 6a1d3110fbc373592c15eade. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"30"}. This record does not assert independent reproduction. ### H2O + OBCACHE-K Accuracy: 75.4. Result ID: 6a1d3110fbc373592c15ead9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"30"}. This record does not assert independent reproduction. ### H2O Accuracy: 75.3. Result ID: 6a1d3110fbc373592c15ead0. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"40"}. This record does not assert independent reproduction. ### TOVA Accuracy: 74.5. Result ID: 6a1d3110fbc373592c15eae2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"Avg."}. This record does not assert independent reproduction. ### H2O + OBCACHE-V Accuracy: 68.6. Result ID: 6a1d3110fbc373592c15ead4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"30"}. This record does not assert independent reproduction. ### H2O + OBCACHE-V&K Accuracy: 67.8. Result ID: 6a1d3110fbc373592c15eae0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"Avg."}. This record does not assert independent reproduction. ### H2O + OBCACHE-K Accuracy: 67.6. Result ID: 6a1d3110fbc373592c15eadb. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"Avg."}. This record does not assert independent reproduction. ### H2O + OBCACHE-V&K Accuracy: 67.5. Result ID: 6a1d3110fbc373592c15eadd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"20"}. This record does not assert independent reproduction. ### H2O Accuracy: 67.1. Result ID: 6a1d3110fbc373592c15eacf. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"30"}. This record does not assert independent reproduction. ### H2O + OBCACHE-K Accuracy: 65.9. Result ID: 6a1d3110fbc373592c15ead8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"20"}. This record does not assert independent reproduction. ### TOVA + OBCACHE-V&K Accuracy: 65.5. Result ID: 6a1d3110fbc373592c15eae3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"10"}. This record does not assert independent reproduction. ### TOVA Accuracy: 61.8. Result ID: 6a1d3110fbc373592c15eae1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"10"}. This record does not assert independent reproduction. ### H2O + OBCACHE-V Accuracy: 59.9. Result ID: 6a1d3110fbc373592c15ead6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"Avg."}. This record does not assert independent reproduction. ### H2O Accuracy: 57.5. Result ID: 6a1d3110fbc373592c15ead1. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"Avg."}. This record does not assert independent reproduction. ### H2O + OBCACHE-V Accuracy: 57.1. Result ID: 6a1d3110fbc373592c15ead3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"20"}. This record does not assert independent reproduction. ### AdaKV + OBCACHE-V&K Accuracy: 55.2. Result ID: 6a1d3110fbc373592c15eae8. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Agnostic","KV Budget (%)":"Avg."}. This record does not assert independent reproduction. ### H2O Accuracy: 54.2. Result ID: 6a1d3110fbc373592c15eace. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"20"}. This record does not assert independent reproduction. ### H2O + OBCACHE-K Accuracy: 47.7. Result ID: 6a1d3110fbc373592c15ead7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"10"}. This record does not assert independent reproduction. ### H2O + OBCACHE-V&K Accuracy: 46.3. Result ID: 6a1d3110fbc373592c15eadc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"10"}. This record does not assert independent reproduction. ### H2O + OBCACHE-V Accuracy: 37.7. Result ID: 6a1d3110fbc373592c15ead2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"10"}. This record does not assert independent reproduction. ### H2O Accuracy: 33.5. Result ID: 6a1d3110fbc373592c15eacd. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Aware","KV Budget (%)":"10"}. This record does not assert independent reproduction. ### SnapKV + OBCACHE-K Accuracy: 24.1. Result ID: 6a1d3110fbc373592c15eae6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Agnostic","KV Budget (%)":"10"}. This record does not assert independent reproduction. ### AdaKV Accuracy: 21.5. Result ID: 6a1d3110fbc373592c15eae7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Agnostic","KV Budget (%)":"10"}. This record does not assert independent reproduction. ### SnapKV Accuracy: 19.4. Result ID: 6a1d3110fbc373592c15eae5. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"LLaMA-3.1-8B","Evaluation Protocol":"Q-Agnostic","KV Budget (%)":"10"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.