Long-context retrieval and synthetic reasoning on RULER
83.01AccuracyGemini-3.0-pro
Evaluation results
| Method | Links | |
|---|---|---|
| Gemini-3.0-pro2026.03 | 83.01 | |
| Deepseek-R1-Distill-Qwen-14B + TableLong2026.03 | 80.72 | |
| Deepseek-R1-Distill-Qwen-32B + TableLong2026.03 | 66.41 | |
| Qwen2.5-32B-Instruct + TableLong2026.03 | 62.65 | |
| Deepseek-R1-Distill-Qwen-14B2026.03 | 61.28 | |
| Deepseek-R1-Distill-Qwen-32B2026.03 | 58.11 | |
| Qwen3-32B + TableLong2026.03 | 53.55 | |
| HalfRoPEModel Size=Llama-3B, Sequence Length=4k2026.02 | 51.28 | |
| Qwen3-32B2026.03 | 50.86 | |
| Qwen-Long-L12026.03 | 47.7 | |
| RoPEModel Size=Llama-3B, Sequence Length=4k2026.02 | 46.19 | |
| Qwen2.5-32B-Instruct2026.03 | 45.25 | |
| YaRNModel Size=Llama-3B, Sequence Length=8k2026.02 | 45.09 | |
| RoPE-IDModel Size=Llama-3B, Sequence Length=4k2026.02 | 44.86 | |
| RoPE-ID (scaling)Model Size=Llama-3B, Sequence Length=4k2026.02 | 44.86 | |
| YaRNModel Size=Llama-3B, Sequence Length=4k2026.02 | 43.9 | |
| RoPE-ID (scaling)Model Size=Llama-3B, Sequence Length=8k2026.02 | 43.39 | |
| HalfRoPEModel Size=Llama-1B, Sequence Length=4k2026.02 | 43.07 | |
| Deepseek-v3.12026.03 | 42.97 | |
| RoPE-ID (scaling)Model Size=Llama-3B, Sequence Length=16k2026.02 | 42 | |
| YaRNModel Size=Llama-1B, Sequence Length=4k2026.02 | 40.24 | |
| YaRNModel Size=Llama-3B, Sequence Length=16k2026.02 | 40.14 | |
| RoPEModel Size=Llama-1B, Sequence Length=4k2026.02 | 39.72 | |
| RoPE-IDModel Size=Llama-1B, Sequence Length=4k2026.02 | 39.15 | |
| RoPE-ID (scaling)Model Size=Llama-1B, Sequence Length=4k2026.02 | 39.15 | |
| RoPE-IDModel Size=Llama-3B, Sequence Length=8k2026.02 | 37.51 | |
| RoPE-ID (scaling)Model Size=Llama-1B, Sequence Length=8k2026.02 | 35.64 | |
| YaRNModel Size=Llama-1B, Sequence Length=8k2026.02 | 35.55 | |
| RoPE-ID (scaling)Model Size=Llama-1B, Sequence Length=16k2026.02 | 30.83 | |
| YaRNModel Size=Llama-1B, Sequence Length=16k2026.02 | 30.25 | |
| RoPE-IDModel Size=Llama-1B, Sequence Length=8k2026.02 | 29.71 | |
| High frequencyModel Size=Llama-3B, Sequence Length=4k2026.02 | 28.02 | |
| RoPE-IDModel Size=Llama-3B, Sequence Length=16k2026.02 | 21.95 | |
| High frequencyModel Size=Llama-1B, Sequence Length=4k2026.02 | 16.04 | |
| High frequencyModel Size=Llama-3B, Sequence Length=8k2026.02 | 14.31 | |
| RoPE-IDModel Size=Llama-1B, Sequence Length=16k2026.02 | 14.29 | |
| High frequencyModel Size=Llama-1B, Sequence Length=8k2026.02 | 7.6 | |
| High frequencyModel Size=Llama-3B, Sequence Length=16k2026.02 | 5.14 | |
| High frequencyModel Size=Llama-1B, Sequence Length=16k2026.02 | 2.37 | |
| HalfRoPEModel Size=Llama-3B, Sequence Length=8k2026.02 | 0.4 | |
| HalfRoPEModel Size=Llama-1B, Sequence Length=8k2026.02 | 0.14 | |
| RoPEModel Size=Llama-3B, Sequence Length=8k2026.02 | 0.14 | |
| RoPEModel Size=Llama-1B, Sequence Length=16k2026.02 | 0.03 | |
| HalfRoPEModel Size=Llama-3B, Sequence Length=16k2026.02 | 0.03 | |
| RoPEModel Size=Llama-1B, Sequence Length=8k2026.02 | 0.01 | |
| RoPEModel Size=Llama-3B, Sequence Length=16k2026.02 | 0.01 | |
| HalfRoPEModel Size=Llama-1B, Sequence Length=16k2026.02 | 0 |