Long-context understanding on LongBench (test) (Including Efficiency Metrics)
71.4FewShot PerformanceQwen3-8B
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8BCategory=Naive LLM-FT, Fine-tuned=true2026.01 | 71.4 | 43.6 | 38.2 | 28.3 | — | 63.8 | — | — | — | — | — | — | — | |
| LongLLMLinguaToken constraint=3,000, LLM Engine=GPT-3.5-Turbo, Method Category=Compression-based2023.10 | 70.6 | 40.7 | 46.2 | 27.2 | 53 | 55.2 | 48.8 | 3,283 | 3 | 10 | 1.6 | — | — | |
| LongLLMLinguaTokens constraint=3,000 tokens, Tokens=3283, Ratio=3x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 70.6 | 40.7 | 46.2 | 27.2 | 53 | 55.2 | 48.8 | — | — | — | — | — | — | |
| EFPCTokens constraint=3,000 tokens, Tokens=3415, Ratio=3x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 70.4 | 42.9 | 46.6 | 26.9 | 32.5 | 59.7 | 46.5 | — | — | — | — | — | — | |
| AdmTreeCategory=Compression-based Methods, Model Scale=7B2026.01 | 69.9 | 41.6 | 45.9 | 30.2 | — | 66.8 | — | — | — | — | — | — | — | |
| LongLLMLinguaToken constraint=2,000, LLM Engine=GPT-3.5-Turbo, Method Category=Compression-based2023.10 | 69.8 | 39.9 | 43.2 | 27.4 | 53 | 56.7 | 48.3 | 1,822 | 6 | 6.1 | 2.6 | — | — | |
| LLMLingua-2Tokens constraint=3,000 tokens, Tokens=3392, Ratio=3x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 69.6 | 35.5 | 38.7 | 26.3 | 21.4 | 62.8 | 42.2 | — | — | — | — | — | — | |
| Full CacheModel=LLaMA-3.1-8B-Instruct, Budget=Full2026.06 | 69.45 | — | — | — | — | — | 49.29 | — | — | — | — | 44.38 | 46.54 | |
| MOMENTKVModel=LLaMA-3.1-8B-Instruct, Budget=10242026.06 | 69.43 | — | — | — | — | — | 48.89 | — | — | — | — | 44.28 | 46.46 | |
| LongLLMLinguaTokens constraint=2,000 tokens, Tokens=1809, Ratio=6x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 69.3 | 39 | 42.2 | 27.4 | 53.8 | 56.6 | 48 | — | — | — | — | — | — | |
| MOMENTKVModel=LLaMA-3.1-8B-Instruct, Budget=5122026.06 | 69.13 | — | — | — | — | — | 48.27 | — | — | — | — | 43.41 | 46.17 | |
| Ada-KVModel=LLaMA-3.1-8B-Instruct, Budget=10242026.06 | 68.84 | — | — | — | — | — | 48.3 | — | — | — | — | 43.42 | 46.32 | |
| SnapKVModel=LLaMA-3.1-8B-Instruct, Budget=10242026.06 | 68.6 | — | — | — | — | — | 48.3 | — | — | — | — | 43.5 | 46.11 | |
| Ada-KVModel=LLaMA-3.1-8B-Instruct, Budget=5122026.06 | 68.5 | — | — | — | — | — | 47.55 | — | — | — | — | 41.6 | 45.6 | |
| PyramidKVModel=LLaMA-3.1-8B-Instruct, Budget=10242026.06 | 68.46 | — | — | — | — | — | 48.09 | — | — | — | — | 43.34 | 46.23 | |
| MOMENTKVModel=LLaMA-3.1-8B-Instruct, Budget=2562026.06 | 68.42 | — | — | — | — | — | 47.59 | — | — | — | — | 41.94 | 45.9 | |
| BeaconCategory=Compression-based Methods2026.01 | 68.4 | 40.5 | 40.3 | 26.8 | — | 66.4 | — | — | — | — | — | — | — | |
| GPT-CTokens constraint=3,000 tokens, Tokens=3187, Ratio=3x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 68.4 | 39.9 | 40.6 | 27 | 48.1 | 58.4 | 47.1 | — | — | — | — | — | — | |
| PyramidKVModel=LLaMA-3.1-8B-Instruct, Budget=5122026.06 | 68.23 | — | — | — | — | — | 47.41 | — | — | — | — | 42.58 | 45.55 | |
| LongLLMLingua rkToken constraint=3,000, LLM Engine=GPT-3.5-Turbo, Method Category=Retrieval-based2023.10 | 68.2 | 37.6 | 42.9 | 26.9 | 49.9 | 53.4 | 46.5 | 3,424 | 3 | 8.2 | 1.9 | — | — | |
| Qwen3-4BCategory=Naive LLM-FT, Fine-tuned=true2026.01 | 68 | 36.4 | 39.3 | 28.9 | — | 62.7 | — | — | — | — | — | — | — | |
| H2OModel=LLaMA-3.1-8B-Instruct, Budget=10242026.06 | 67.81 | — | — | — | — | — | 48 | — | — | — | — | 43.4 | 45.84 | |
| GPT-CTokens constraint=2,000 tokens, Tokens=1898, Ratio=5x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 67.7 | 36.3 | 38.5 | 27.1 | 46.3 | 53.1 | 44.8 | — | — | — | — | — | — | |
| SnapKVModel=LLaMA-3.1-8B-Instruct, Budget=5122026.06 | 67.6 | — | — | — | — | — | 47.29 | — | — | — | — | 41.77 | 45.36 | |
| Ada-KVModel=LLaMA-3.1-8B-Instruct, Budget=2562026.06 | 67.58 | — | — | — | — | — | 46.79 | — | — | — | — | 40.23 | 45.41 | |
| EFPCTokens constraint=2,000 tokens, Tokens=1972, Ratio=5x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 67.3 | 41.7 | 42.4 | 25.8 | 27 | 57.6 | 43.6 | — | — | — | — | — | — | |
| LLMLinguaToken constraint=3,000, LLM Engine=GPT-3.5-Turbo, Method Category=Compression-based2023.10 | 67.2 | 31.8 | 37.5 | 26.2 | 8.3 | 53.2 | 37.4 | 3,421 | 3 | 9.2 | 1.7 | — | — | |
| LLMLinguaTokens constraint=3,000 tokens, Tokens=3421, Ratio=3x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 67.2 | 31.8 | 37.5 | 26.2 | 8.3 | 53.2 | 37.4 | — | — | — | — | — | — | |
| SnapKVCategory=Compression-based Methods2026.01 | 67.1 | 38.7 | 37.6 | 26.2 | — | 60.3 | — | — | — | — | — | — | — | |
| Original PromptLLM Engine=GPT-3.5-Turbo2023.10 | 67 | 39.7 | 38.7 | 26.5 | 37.8 | 54.2 | 44 | 10,295 | — | 15.6 | — | — | — | |
| PyramidKVModel=LLaMA-3.1-8B-Instruct, Budget=2562026.06 | 66.94 | — | — | — | — | — | 46.01 | — | — | — | — | 40.71 | 44.34 | |
| MOMENTKVModel=LLaMA-3.1-8B-Instruct, Budget=1282026.06 | 66.73 | — | — | — | — | — | 46.38 | — | — | — | — | 40.15 | 45.11 | |
| LLMLingua-2Tokens constraint=2,000 tokens, Tokens=1954, Ratio=5x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 66.4 | 29.8 | 33.1 | 25.3 | 21.3 | 58.9 | 39.1 | — | — | — | — | — | — | |
| LongLLMLingua rkToken constraint=2,000, LLM Engine=GPT-3.5-Turbo, Method Category=Retrieval-based2023.10 | 66.3 | 37.8 | 41.7 | 26.9 | 53 | 52.4 | 46.3 | 1,960 | 5 | 4.7 | 3.3 | — | — | |
| Full CacheModel=Qwen3-4B-Instruct-2507, Budget=Full2026.06 | 65.52 | — | — | — | — | — | 45.13 | — | — | — | — | 40.25 | 42.18 | |
| Qwen3-4BCategory=Naive LLM2026.01 | 65.5 | 25.2 | 23.8 | 23.1 | — | 55.9 | — | — | — | — | — | — | — | |
| MOMENTKVModel=Qwen3-4B-Instruct-2507, Budget=10242026.06 | 65.48 | — | — | — | — | — | 45.38 | — | — | — | — | 40.24 | 42.15 | |
| Ada-KVModel=Qwen3-4B-Instruct-2507, Budget=10242026.06 | 65.42 | — | — | — | — | — | 45.33 | — | — | — | — | 40.22 | 42.12 | |
| SnapKVModel=Qwen3-4B-Instruct-2507, Budget=10242026.06 | 65.28 | — | — | — | — | — | 45.23 | — | — | — | — | 40.12 | 42.05 | |
| MOMENTKVModel=Qwen3-4B-Instruct-2507, Budget=5122026.06 | 65.25 | — | — | — | — | — | 45.45 | — | — | — | — | 40.18 | 42.05 | |
| VIST2-8BCategory=Ours2026.01 | 65.2 | 45.2 | 49.8 | 30.4 | — | 66.5 | — | — | — | — | — | — | — | |
| H2OModel=Qwen3-4B-Instruct-2507, Budget=10242026.06 | 65.12 | — | — | — | — | — | 44.94 | — | — | — | — | 40.02 | 41.95 | |
| PyramidKVModel=Qwen3-4B-Instruct-2507, Budget=10242026.06 | 65.02 | — | — | — | — | — | 45.07 | — | — | — | — | 39.95 | 41.92 | |
| Ada-KVModel=LLaMA-3.1-8B-Instruct, Budget=1282026.06 | 65 | — | — | — | — | — | 45.03 | — | — | — | — | 37.28 | 44.31 | |
| Ada-KVModel=Qwen3-4B-Instruct-2507, Budget=5122026.06 | 64.92 | — | — | — | — | — | 45.1 | — | — | — | — | 40.05 | 41.88 | |
| MOMENTKVModel=Qwen3-4B-Instruct-2507, Budget=2562026.06 | 64.82 | — | — | — | — | — | 45.23 | — | — | — | — | 39.95 | 41.85 | |
| PyramidKVModel=LLaMA-3.1-8B-Instruct, Budget=1282026.06 | 64.5 | — | — | — | — | — | 44.46 | — | — | — | — | 37.42 | 43.98 | |
| SnapKVModel=Qwen3-4B-Instruct-2507, Budget=5122026.06 | 64.38 | — | — | — | — | — | 44.86 | — | — | — | — | 39.85 | 41.72 | |
| RAGPTokens constraint=3,000 tokens, Tokens=2691, Ratio=4x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 64.2 | 43.6 | 44.7 | 22.7 | 62.5 | 58.4 | 49.3 | — | — | — | — | — | — | |
| MOMENTKVModel=Qwen3-4B-Instruct-2507, Budget=1282026.06 | 64.18 | — | — | — | — | — | 44.59 | — | — | — | — | 38.65 | 40.72 | |
| Ada-KVModel=Qwen3-4B-Instruct-2507, Budget=2562026.06 | 64.15 | — | — | — | — | — | 44.71 | — | — | — | — | 39.68 | 41.42 | |
| PyramidKVModel=Qwen3-4B-Instruct-2507, Budget=5122026.06 | 64.05 | — | — | — | — | — | 44.63 | — | — | — | — | 39.62 | 41.55 | |
| H2OModel=Qwen3-4B-Instruct-2507, Budget=5122026.06 | 63.78 | — | — | — | — | — | 44.42 | — | — | — | — | 39.55 | 41.42 | |
| SnapKVModel=LLaMA-3.1-8B-Instruct, Budget=2562026.06 | 63.71 | — | — | — | — | — | 45.5 | — | — | — | — | 39.24 | 44.47 | |
| H2OModel=LLaMA-3.1-8B-Instruct, Budget=5122026.06 | 63.66 | — | — | — | — | — | 46.08 | — | — | — | — | 40.83 | 44.5 | |
| SnapKVModel=Qwen3-4B-Instruct-2507, Budget=2562026.06 | 63.52 | — | — | — | — | — | 44.38 | — | — | — | — | 39.22 | 41.15 | |
| Ada-KVModel=Qwen3-4B-Instruct-2507, Budget=1282026.06 | 63.48 | — | — | — | — | — | 43.64 | — | — | — | — | 37.52 | 39.85 | |
| SBERTToken constraint=3,000, LLM Engine=GPT-3.5-Turbo, Method Category=Retrieval-based2023.10 | 63.4 | 35.3 | 37.4 | 26.7 | 51 | 34.5 | 41.4 | 3,399 | 3 | 7.7 | 2 | — | — | |
| OpenAIToken constraint=3,000, LLM Engine=GPT-3.5-Turbo, Method Category=Retrieval-based2023.10 | 63.4 | 34.5 | 38.6 | 26.8 | 49.6 | 37.6 | 41.7 | 3,421 | 3 | 13.3 | 1.2 | — | — | |
| SBERTTokens constraint=3,000 tokens, Tokens=3399, Ratio=3x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 63.4 | 35.3 | 37.4 | 26.7 | 51 | 34.5 | 41.4 | — | — | — | — | — | — | |
| OpenAITokens constraint=3,000 tokens, Tokens=3421, Ratio=3x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 63.4 | 34.5 | 38.6 | 26.8 | 49.6 | 37.6 | 41.7 | — | — | — | — | — | — | |
| PyramidKVModel=Qwen3-4B-Instruct-2507, Budget=2562026.06 | 63.1 | — | — | — | — | — | 44.11 | — | — | — | — | 38.85 | 40.88 | |
| SnapKVModel=Qwen3-4B-Instruct-2507, Budget=1282026.06 | 62.85 | — | — | — | — | — | 43.24 | — | — | — | — | 36.9 | 39.48 | |
| PyramidKVModel=Qwen3-4B-Instruct-2507, Budget=1282026.06 | 62.15 | — | — | — | — | — | 42.9 | — | — | — | — | 36.45 | 39.22 | |
| RAGPTokens constraint=2,000 tokens, Tokens=1793, Ratio=6x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 62.1 | 41.9 | 42.7 | 22.4 | 62 | 57.1 | 48.1 | — | — | — | — | — | — | |
| H2OModel=Qwen3-4B-Instruct-2507, Budget=2562026.06 | 61.85 | — | — | — | — | — | 43.16 | — | — | — | — | 37.15 | 40.02 | |
| CPC*Tokens constraint=3,000 tokens, Tokens=3327, Ratio=3x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 61.7 | 41.2 | 46.1 | 23.7 | 55.8 | 50.9 | 46.5 | — | — | — | — | — | — | |
| H2OModel=LLaMA-3.1-8B-Instruct, Budget=2562026.06 | 61.26 | — | — | — | — | — | 44.28 | — | — | — | — | 37.78 | 43.18 | |
| LLMLinguaToken constraint=2,000, LLM Engine=GPT-3.5-Turbo, Method Category=Compression-based2023.10 | 61.2 | 22.4 | 32.1 | 24.5 | 10.4 | 56.8 | 34.6 | 1,950 | 5 | 5.9 | 2.6 | — | — | |
| LLMLinguaTokens constraint=2,000 tokens, Tokens=1950, Ratio=5x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 61.2 | 22.4 | 32.1 | 24.5 | 10.4 | 56.8 | 34.6 | — | — | — | — | — | — | |
| VIST2-4BCategory=Ours2026.01 | 60.5 | 41.5 | 44.2 | 28.1 | — | 64.9 | — | — | — | — | — | — | — | |
| Qwen3-8BCategory=Naive LLM2026.01 | 60.3 | 29.1 | 24.6 | 59.4 | — | 42.5 | — | — | — | — | — | — | — | |
| SnapKVModel=LLaMA-3.1-8B-Instruct, Budget=1282026.06 | 59.57 | — | — | — | — | — | 43.21 | — | — | — | — | 35.98 | 42.85 | |
| CPC*Tokens constraint=2,000 tokens, Tokens=1844, Ratio=5x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 58.8 | 36.9 | 44.8 | 23.3 | 56.5 | 45.6 | 44.3 | — | — | — | — | — | — | |
| H2OModel=LLaMA-3.1-8B-Instruct, Budget=1282026.06 | 58.42 | — | — | — | — | — | 42.21 | — | — | — | — | 34.44 | 41.83 | |
| BM25Token constraint=3,000, LLM Engine=GPT-3.5-Turbo, Method Category=Retrieval-based2023.10 | 57.9 | 32.3 | 34.3 | 25.3 | 45.1 | 48.9 | 40.6 | 3,417 | 3 | 7.5 | 2.1 | — | — | |
| BM25Tokens constraint=3,000 tokens, Tokens=3417, Ratio=3x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 57.9 | 32.3 | 34.3 | 25.3 | 45.1 | 48.9 | 40.6 | — | — | — | — | — | — | |
| H2OModel=Qwen3-4B-Instruct-2507, Budget=1282026.06 | 57.2 | — | — | — | — | — | 40.86 | — | — | — | — | 33.82 | 38.15 | |
| LongLLMLinguaCategory=Compression-based Methods2026.01 | 55.9 | 24.7 | 20.3 | 26.3 | — | 50.1 | — | — | — | — | — | — | — | |
| VIST2-0.6BCategory=Ours2026.01 | 54.1 | 36.8 | 39.5 | 25.2 | — | 58.4 | — | — | — | — | — | — | — | |
| BM25Category=Retrieval-based Methods2026.01 | 54 | 28.8 | 31.1 | 23.8 | — | 32 | — | — | — | — | — | — | — | |
| SBERTCategory=Retrieval-based Methods2026.01 | 50.4 | 18.4 | 22.9 | 21.6 | — | 34.6 | — | — | — | — | — | — | — | |
| Zero-shotLLM Engine=GPT-3.5-Turbo2023.10 | 40.7 | 15.6 | 31.3 | 15.6 | 1.6 | 36.2 | 23.5 | 214 | 48 | 1.6 | 9.8 | — | — | |
| OpenAIToken constraint=2,000, LLM Engine=GPT-3.5-Turbo, Method Category=Retrieval-based2023.10 | 32.4 | 34.3 | 36.3 | 24.7 | 26.3 | 24.8 | 29.8 | 1,991 | 5 | 10.4 | 1.5 | — | — | |
| OpenAITokens constraint=2,000 tokens, Tokens=1991, Ratio=5x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 32.4 | 34.3 | 36.3 | 24.7 | 26.3 | 24.8 | 29.8 | — | — | — | — | — | — | |
| Selective-ContextToken constraint=3,000, LLM Engine=GPT-3.5-Turbo, Method Category=Compression-based2023.10 | 23.8 | 23.3 | 39.2 | 25 | 27.5 | 53.1 | 32 | 3,328 | 3 | 50.6 | 0.3 | — | — | |
| Selective-ContextTokens constraint=3,000 tokens, Tokens=3328, Ratio=3x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 23.8 | 23.3 | 39.2 | 25 | 27.5 | 53.1 | 32 | — | — | — | — | — | — | |
| SBERTToken constraint=2,000, LLM Engine=GPT-3.5-Turbo, Method Category=Retrieval-based2023.10 | 23.5 | 33.8 | 35.9 | 25.9 | 18 | 17.8 | 25.8 | 1,947 | 5 | 4.8 | 3.4 | — | — | |
| SBERTTokens constraint=2,000 tokens, Tokens=1947, Ratio=5x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 23.5 | 33.8 | 35.9 | 25.9 | 18 | 17.8 | 25.8 | — | — | — | — | — | — | |
| OpenAICategory=Retrieval-based Methods2026.01 | 22.9 | 30 | 19.1 | 23.8 | — | 51.6 | — | — | — | — | — | — | — | |
| BM25Token constraint=2,000, LLM Engine=GPT-3.5-Turbo, Method Category=Retrieval-based2023.10 | 19.5 | 30.1 | 29.4 | 21.2 | 12.4 | 29.1 | 23.6 | 1,985 | 5 | 4.6 | 3.4 | — | — | |
| BM25Tokens constraint=2,000 tokens, Tokens=1985, Ratio=5x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 19.5 | 30.1 | 29.4 | 21.2 | 12.4 | 29.1 | 23.6 | — | — | — | — | — | — | |
| Selective-ContextToken constraint=2,000, LLM Engine=GPT-3.5-Turbo, Method Category=Compression-based2023.10 | 15.7 | 16.2 | 34.8 | 24.4 | 8.4 | 49.2 | 24.8 | 1,925 | 5 | 47.1 | 0.3 | — | — | |
| Selective-ContextTokens constraint=2,000 tokens, Tokens=1925, Ratio=5x, Evaluator LLM=GPT-3.5-Turbo2026.05 | 15.7 | 16.2 | 34.8 | 24.4 | 8.4 | 49.2 | 24.8 | — | — | — | — | — | — |