Long-context Question Answering on LongBench V2
56.77Overall AccuracyGemini-2.5-Flash-Thinking
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini-2.5-Flash-ThinkingBackbone=Gemini-2.5-Flash-Thinking, Category=Frontier Models2026.01 | 56.77 | 51.43 | 55.2 | 58 | 66.67 | — | — | — | — | — | — | — | — | — | — | |
| LONGPASBackbone=Qwen3-30B-A3B-Thinking, Category=Reasoning Models, Training Strategy=LONGPAS2026.01 | 54.27 | 47.43 | 54.4 | 60 | 76.92 | — | — | — | — | — | — | — | — | — | — | |
| LONGPASBackbone=Qwen3-30B-A3B-Instruct, Category=Instruct Models, Training Strategy=LONGPAS2026.01 | 49.11 | 46.29 | 42.4 | 56 | 71.79 | — | — | — | — | — | — | — | — | — | — | |
| Qwen3-30B-A3B-ThinkingBackbone=Qwen3-30B-A3B-Thinking, Category=Reasoning Models, Training Strategy=Vanilla2026.01 | 48.31 | 44 | 44 | 46 | 64.1 | — | — | — | — | — | — | — | — | — | — | |
| RLVRBackbone=Qwen3-30B-A3B-Instruct, Category=Instruct Models, Training Strategy=RLVR2026.01 | 47.91 | 50.29 | 50.4 | 42 | 56.41 | — | — | — | — | — | — | — | — | — | — | |
| GPT-OSS-120BBackbone=GPT-OSS-120B, Category=Frontier Models2026.01 | 47.01 | 44.57 | 43.2 | 53.06 | 61.54 | — | — | — | — | — | — | — | — | — | — | |
| RLVRBackbone=Qwen3-30B-A3B-Thinking, Category=Reasoning Models, Training Strategy=RLVR2026.01 | 44.53 | 40 | 46.4 | 50 | 51.28 | — | — | — | — | — | — | — | — | — | — | |
| Qwen3-30B-A3B-InstructBackbone=Qwen3-30B-A3B-Instruct, Category=Instruct Models, Training Strategy=Vanilla2026.01 | 44.43 | 42.43 | 38.4 | 52 | 62.82 | — | — | — | — | — | — | — | — | — | — | |
| GPT5-NanoBackbone=GPT5-Nano, Category=Frontier Models2026.01 | 43.74 | 44 | 39.2 | 50 | 46.15 | — | — | — | — | — | — | — | — | — | — | |
| GPT-OSS-20BBackbone=GPT-OSS-20B, Category=Frontier Models2026.01 | 43.37 | 38.51 | 40.8 | 56 | 61.54 | — | — | — | — | — | — | — | — | — | — | |
| LONGPASBackbone=Qwen3-4B-Instruct, Category=Instruct Models, Training Strategy=LONGPAS2026.01 | 42.94 | 38.14 | 40.6 | 48 | 63.46 | — | — | — | — | — | — | — | — | — | — | |
| MemSearch-o12026.04 | 42.31 | — | — | — | — | 43.48 | 40.43 | 42.42 | 48.65 | 36.59 | — | — | — | — | — | |
| LONGPASBackbone=Qwen3-4B-Thinking, Category=Reasoning Models, Training Strategy=LONGPAS2026.01 | 42.3 | 40.29 | 40 | 43 | 62.82 | — | — | — | — | — | — | — | — | — | — | |
| RLVRBackbone=Qwen3-4B-Instruct, Category=Instruct Models, Training Strategy=RLVR2026.01 | 42.1 | 39.14 | 37 | 49.5 | 60.9 | — | — | — | — | — | — | — | — | — | — | |
| RLVRBackbone=Qwen3-4B-Thinking, Category=Reasoning Models, Training Strategy=RLVR2026.01 | 41.75 | 37.14 | 40.8 | 46 | 66.67 | — | — | — | — | — | — | — | — | — | — | |
| Qwen3-4B-ThinkingBackbone=Qwen3-4B-Thinking, Category=Reasoning Models, Training Strategy=Vanilla2026.01 | 40.46 | 37 | 35.6 | 41.5 | 60.26 | — | — | — | — | — | — | — | — | — | — | |
| A-Mem2026.04 | 39.33 | — | — | — | — | 39.13 | 40.43 | 42.42 | 40.54 | 34.15 | — | — | — | — | — | |
| Qwen3-4B-InstructBackbone=Qwen3-4B-Instruct, Category=Instruct Models, Training Strategy=Vanilla2026.01 | 37.28 | 36 | 30.6 | 36.5 | 60.26 | — | — | — | — | — | — | — | — | — | — | |
| Amber2026.04 | 35.85 | — | — | — | — | 39.13 | 39.36 | 36.36 | 35.14 | 29.27 | — | — | — | — | — | |
| LONGPASBackbone=Qwen2.5-7B-Instruct, Category=Instruct Models, Training Strategy=LONGPAS2026.01 | 33.7 | 39.57 | 28.4 | 29.5 | 31.41 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-7B-InstructBackbone=Qwen2.5-7B-Instruct, Category=Instruct Models, Training Strategy=Vanilla2026.01 | 33.6 | 36.71 | 28.2 | 32 | 35.9 | — | — | — | — | — | — | — | — | — | — | |
| KV-CATKeep ratio=50%, Compression Method=gradient-based2026.05 | 31.2 | — | — | — | — | — | 26.6 | 42.4 | — | — | 30 | 26.7 | 33.3 | 35 | 33.3 | |
| RLVRBackbone=Qwen2.5-7B-Instruct, Category=Instruct Models, Training Strategy=RLVR2026.01 | 31.01 | 34.43 | 29 | 30 | 40.38 | — | — | — | — | — | — | — | — | — | — | |
| Search-o1 (Refined)2026.04 | 30.96 | — | — | — | — | 30.43 | 27.27 | 32.98 | 32.43 | 31.71 | — | — | — | — | — | |
| KV-CATKeep ratio=20%, Compression Method=gradient-based2026.05 | 30.8 | — | — | — | — | — | 27.7 | 45.5 | — | — | 30 | 26.7 | 28.6 | 35 | 22.2 | |
| KV-CATKeep ratio=10%, Compression Method=gradient-based2026.05 | 30.3 | — | — | — | — | — | 26.6 | 48.5 | — | — | 30 | 33.3 | 19 | 30 | 27.8 | |
| LONGPASBackbone=LLaMA3.1-8B-Instruct, Category=Instruct Models, Training Strategy=LONGPAS2026.01 | 29.62 | 30.14 | 29.8 | 32 | 34.62 | — | — | — | — | — | — | — | — | — | — | |
| RLVRBackbone=LLaMA3.1-8B-Instruct, Category=Instruct Models, Training Strategy=RLVR2026.01 | 28.93 | 31.86 | 29.2 | 29 | 32.69 | — | — | — | — | — | — | — | — | — | — | |
| Direct RAG2026.04 | 28.91 | — | — | — | — | 30.43 | 33.33 | 24.24 | 29.73 | 26.83 | — | — | — | — | — | |
| LLaMA3.1-8B-InstructBackbone=LLaMA3.1-8B-Instruct, Category=Instruct Models, Training Strategy=Vanilla2026.01 | 27.93 | 28.86 | 26.2 | 27 | 32.05 | — | — | — | — | — | — | — | — | — | — | |
| BaseKeep ratio=50%, Compression Method=gradient-based2026.05 | 25.3 | — | — | — | — | — | 21.3 | 27.3 | — | — | 30 | 20 | 38.1 | 15 | 38.9 | |
| BaseKeep ratio=10%, Compression Method=gradient-based2026.05 | 22.2 | — | — | — | — | — | 18.1 | 27.3 | — | — | 20 | 33.3 | 28.6 | 30 | 11.1 | |
| BaseKeep ratio=20%, Compression Method=gradient-based2026.05 | 22.2 | — | — | — | — | — | 19.1 | 27.3 | — | — | 25 | 26.7 | 28.6 | 15 | 22.2 |