# Long-context language understanding on RULER 16K Source: [Long-context language understanding on RULER 16K]() Collection index: [llms.txt]() Data updated: 2026-06-01T09:27:01.703Z Evaluation of model performance on the RULER 16K dataset for long-context language understanding, reporting scores for CWE, FWE, NIAH subtasks (MK1-MK3, MQ, MV, S1-S3), QA tasks (1 & 2), VT, and overall average. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Long-context language understanding | | Dataset (including any stated split) | RULER 16K | | Metric | CWE Score | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 18 of 18 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [CWE Score](): Higher is better; unit: PERCENT. - [FWE Score](): Higher is better; unit: PERCENT. - [NIAH MK1 Score](): Higher is better; unit: PERCENT. - [NIAH MK2 Score](): Higher is better; unit: PERCENT. - [NIAH MK3 Score](): Higher is better; unit: PERCENT. - [NIAH MQ Score](): Higher is better; unit: PERCENT. - [NIAH MV Score](): Higher is better; unit: PERCENT. - [NIAH S1 Score](): Higher is better; unit: PERCENT. - [NIAH S2 Score](): Higher is better; unit: PERCENT. - [NIAH S3 Score](): Higher is better; unit: PERCENT. - [QA 1 Score](): Higher is better; unit: PERCENT. - [QA 2 Score](): Higher is better; unit: PERCENT. - [VT Score](): Higher is better; unit: PERCENT. - [Overall Average Score](): Higher is better; unit: PERCENT. - [Tasks Better Than Baseline Count](): Higher is better; unit: SCALAR. ## Recorded results | Method | CWE Score | Result date | Report paper | | --- | --- | --- | --- | | Full Cache | 89.28 | Not recorded | 6a1d1227dbd7150506f84047 | | Full Cache | 82.64 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV | 77.66 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. GRKV | 77.4 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. GRKV | 70.38 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV | 68.32 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. D2O | 65.64 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. D2O | 59.96 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. CaM | 43.3 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. CaM | 42.42 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. GRKV | 42.22 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV | 41.9 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV | 22.68 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. GRKV | 20.3 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. D2O | 12.9 | Not recorded | 6a1d1227dbd7150506f84047 | | CriticalKV w. CaM | 4.94 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. D2O | 3.9 | Not recorded | 6a1d1227dbd7150506f84047 | | SnapKV w. CaM | 3.58 | Not recorded | 6a1d1227dbd7150506f84047 | ### Full Cache CWE Score: 89.28. Result ID: 6a1d504afbc373592c16d837. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Cache Budget":"Full"}. This record does not assert independent reproduction. ### Full Cache CWE Score: 82.64. Result ID: 6a1d504afbc373592c16d840. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Cache Budget":"Full"}. This record does not assert independent reproduction. ### CriticalKV CWE Score: 77.66. Result ID: 6a1d504afbc373592c16d845. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Cache Budget":"20%"}. This record does not assert independent reproduction. ### CriticalKV w. GRKV CWE Score: 77.4. Result ID: 6a1d504afbc373592c16d848. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Cache Budget":"20%"}. This record does not assert independent reproduction. ### SnapKV w. GRKV CWE Score: 70.38. Result ID: 6a1d504afbc373592c16d844. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Cache Budget":"20%"}. This record does not assert independent reproduction. ### SnapKV CWE Score: 68.32. Result ID: 6a1d504afbc373592c16d841. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Cache Budget":"20%"}. This record does not assert independent reproduction. ### CriticalKV w. D2O CWE Score: 65.64. Result ID: 6a1d504afbc373592c16d847. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Cache Budget":"20%"}. This record does not assert independent reproduction. ### SnapKV w. D2O CWE Score: 59.96. Result ID: 6a1d504afbc373592c16d843. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Cache Budget":"20%"}. This record does not assert independent reproduction. ### CriticalKV w. CaM CWE Score: 43.3. Result ID: 6a1d504afbc373592c16d846. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Cache Budget":"20%"}. This record does not assert independent reproduction. ### SnapKV w. CaM CWE Score: 42.42. Result ID: 6a1d504afbc373592c16d842. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Mistral-7B-Instruct-v0.3","Cache Budget":"20%"}. This record does not assert independent reproduction. ### CriticalKV w. GRKV CWE Score: 42.22. Result ID: 6a1d504afbc373592c16d83f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Cache Budget":"20%"}. This record does not assert independent reproduction. ### CriticalKV CWE Score: 41.9. Result ID: 6a1d504afbc373592c16d83c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Cache Budget":"20%"}. This record does not assert independent reproduction. ### SnapKV CWE Score: 22.68. Result ID: 6a1d504afbc373592c16d838. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Cache Budget":"20%"}. This record does not assert independent reproduction. ### SnapKV w. GRKV CWE Score: 20.3. Result ID: 6a1d504afbc373592c16d83b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Cache Budget":"20%"}. This record does not assert independent reproduction. ### CriticalKV w. D2O CWE Score: 12.9. Result ID: 6a1d504afbc373592c16d83e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Cache Budget":"20%"}. This record does not assert independent reproduction. ### CriticalKV w. CaM CWE Score: 4.94. Result ID: 6a1d504afbc373592c16d83d. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Cache Budget":"20%"}. This record does not assert independent reproduction. ### SnapKV w. D2O CWE Score: 3.9. Result ID: 6a1d504afbc373592c16d83a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Cache Budget":"20%"}. This record does not assert independent reproduction. ### SnapKV w. CaM CWE Score: 3.58. Result ID: 6a1d504afbc373592c16d839. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B-Instruct","Cache Budget":"20%"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.