# Long-context language understanding on RULER 16K 1.0 (test) Source: [Long-context language understanding on RULER 16K 1.0 (test)]() Collection index: [llms.txt]() Data updated: 2026-06-01T09:27:01.637Z Evaluation of model performance on the test split of the RULER dataset configured for a 16,000 token context length (Version 1.0), reporting detailed scores across CWE, FWE, various NIAH components, and QA subtasks. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Long-context language understanding | | Dataset (including any stated split) | RULER 16K 1.0 (test) | | Metric | CWE Score | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 18 of 18 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [CWE Score](): Higher is better; unit: PERCENT. - [FWE Score](): Higher is better; unit: PERCENT. - [NIAH MK1 Score](): Higher is better; unit: PERCENT. - [NIAH MK2 Score](): Higher is better; unit: PERCENT. - [NIAH MK3 Score](): Higher is better; unit: PERCENT. - [NIAH MQ Score](): Higher is better; unit: PERCENT. - [NIAH MV Score](): Higher is better; unit: PERCENT. - [NIAH S1 Score](): Higher is better; unit: PERCENT. - [NIAH S2 Score](): Higher is better; unit: PERCENT. - [NIAH S3 Score](): Higher is better; unit: PERCENT. - [QA 1 Score](): Higher is better; unit: PERCENT. - [QA 2 Score](): Higher is better; unit: PERCENT. - [VT Score](): Higher is better; unit: PERCENT. - [Average Score](): Higher is better; unit: PERCENT. - [Models Better Than Baseline Count](): Higher is better; unit: SCALAR. ## Recorded results | Method | CWE Score | Result date | Report paper | | --- | --- | --- | --- | | Full Cache | 89.28 | Not recorded | 6a1d1227dbd7150506f84047 | | Full Cache | 82.64 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. GRKV | 59.84 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV | 59.1 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. GRKV | 47.32 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV | 45.98 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. D2O | 30.74 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. D2O | 22.78 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. CaM | 10.32 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV | 9.72 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. GRKV | 8.8 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV | 3.24 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. CaM | 2.06 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. GRKV | 1.78 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. D2O | 0.28 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. CaM | 0.12 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. CaM | 0.1 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. D2O | 0.02 | Not recorded | 6a1d1227dbd7150506f84047 | ### Full Cache CWE Score: 89.28. Result ID: 6a1d504afbc373592c16d80f. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Context Length":"16K","Cache Budget":"100%"}. This record does not assert independent reproduction. ### Full Cache CWE Score: 82.64. Result ID: 6a1d504afbc373592c16d818. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Context Length":"16K","Cache Budget":"100%"}. This record does not assert independent reproduction. ### CriticalKV w. GRKV CWE Score: 59.84. Result ID: 6a1d504afbc373592c16d820. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### CriticalKV CWE Score: 59.1. Result ID: 6a1d504afbc373592c16d81d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### SnapKV w. GRKV CWE Score: 47.32. Result ID: 6a1d504afbc373592c16d81c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### SnapKV CWE Score: 45.98. Result ID: 6a1d504afbc373592c16d819. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### CriticalKV w. D2O CWE Score: 30.74. Result ID: 6a1d504afbc373592c16d81f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### SnapKV w. D2O CWE Score: 22.78. Result ID: 6a1d504afbc373592c16d81b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### SnapKV w. CaM CWE Score: 10.32. Result ID: 6a1d504afbc373592c16d81a. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### CriticalKV CWE Score: 9.72. Result ID: 6a1d504afbc373592c16d814. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### CriticalKV w. GRKV CWE Score: 8.8. Result ID: 6a1d504afbc373592c16d817. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### SnapKV CWE Score: 3.24. Result ID: 6a1d504afbc373592c16d810. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### CriticalKV w. CaM CWE Score: 2.06. Result ID: 6a1d504afbc373592c16d81e. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### SnapKV w. GRKV CWE Score: 1.78. Result ID: 6a1d504afbc373592c16d813. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### CriticalKV w. D2O CWE Score: 0.28. Result ID: 6a1d504afbc373592c16d816. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### SnapKV w. CaM CWE Score: 0.12. Result ID: 6a1d504afbc373592c16d811. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### CriticalKV w. CaM CWE Score: 0.1. Result ID: 6a1d504afbc373592c16d815. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. ### SnapKV w. D2O CWE Score: 0.02. Result ID: 6a1d504afbc373592c16d812. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Context Length":"16K","Cache Budget":"10%"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.