Retrieval on RULER 128K context
66.71Accuracygpt-oss-puzzle-88B
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| gpt-oss-puzzle-88BReasoning effort=Medium, KV cache precision=BF162026.02 | 66.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| gpt-oss-puzzle-88BReasoning effort=Low, KV cache precision=BF162026.02 | 66.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| gpt-oss-puzzle-88BReasoning effort=High, KV cache precision=BF162026.02 | 59.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| gpt-oss-puzzle-88BReasoning effort=Low, KV cache precision=FP82026.02 | 56.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| gpt-oss-puzzle-88BReasoning effort=High, KV cache precision=FP82026.02 | 56.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| gpt-oss-120BReasoning effort=Medium, KV cache precision=BF162026.02 | 55.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| gpt-oss-puzzle-88BReasoning effort=Medium, KV cache precision=FP82026.02 | 53.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| gpt-oss-120BReasoning effort=Low, KV cache precision=BF162026.02 | 52.61 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| gpt-oss-120BReasoning effort=High, KV cache precision=BF162026.02 | 50.89 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| gpt-oss-120BReasoning effort=Low, KV cache precision=FP82026.02 | 47.28 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| gpt-oss-120BReasoning effort=High, KV cache precision=FP82026.02 | 45.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| gpt-oss-120BReasoning effort=Medium, KV cache precision=FP82026.02 | 45.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DenseBackbone Model=Llama-3.1-8B-Instruct, Context Length=128K, Token Budget=Full Attention2025.06 | — | 100 | 97.92 | 100 | 97.92 | 77.08 | 94.27 | 96.09 | 51.04 | 75.35 | 78.12 | 40.62 | 82.68 | — | |
| H2OBackbone Model=Llama-3.1-8B-Instruct, Context Length=128K, Token Budget=2048 (1.56%)2025.06 | — | 36.46 | 88.54 | 3.12 | 4.17 | 2.08 | 1.82 | 1.56 | 23.33 | 48.61 | 48.96 | 33.33 | 18.68 | — | |
| HATABackbone Model=Llama-3.1-8B-Instruct, Context Length=128K, Token Budget=2048 (1.56%)2025.06 | — | 98.96 | 98.96 | 100 | 96.88 | 69.79 | 89.06 | 94.53 | 50.21 | 71.18 | 76.04 | 40.62 | 80.57 | — | |
| LokiBackbone Model=Llama-3.1-8B-Instruct, Context Length=128K, Token Budget=2048 (1.56%)2025.06 | — | 98.96 | 93.75 | 96.88 | 96.88 | 59.38 | 91.67 | 94.79 | 50 | 57.99 | 76.04 | 35.29 | 77.8 | — | |
| MagicPIGBackbone Model=Llama-3.1-8B-Instruct, Context Length=128K, Token Budget=2-3%2025.06 | — | 94.79 | 4.17 | 51.04 | 65.62 | 20.83 | 44.79 | 57.81 | 41.67 | 57.99 | 67.71 | 38.54 | 55.51 | — | |
| MrRoPE-ProModel=Qwen2.5-3B-Instruct, Scale Setting=32K → 128K, Extension Method=MrRoPE-Pro2026.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | 53.2 | |
| QuestBackbone Model=Llama-3.1-8B-Instruct, Context Length=128K, Token Budget=2048 (1.56%)2025.06 | — | 100 | 69.79 | 47.92 | 97.35 | 53.12 | 78.91 | 90.1 | 61.25 | 63.19 | 73.96 | 38.54 | 72.55 | — | |
| SnapKVBackbone Model=Llama-3.1-8B-Instruct, Context Length=128K, Token Budget=2048 (1.56%)2025.06 | — | 98.96 | 98.96 | 8.33 | 89.58 | 3.12 | 13.28 | 52.34 | 52.29 | 38.89 | 78.12 | 39.58 | 51.18 | — | |
| StreamingLLMBackbone Model=Llama-3.1-8B-Instruct, Context Length=128K, Token Budget=2048 (1.56%)2025.06 | — | 1.04 | 2.08 | 3.12 | 5.21 | 3.12 | 4.69 | 5.73 | 0.62 | 67.7 | 51.04 | 34.38 | 16.44 | — | |
| YaRNModel=Qwen2.5-3B-Instruct, Scale Setting=32K → 128K, Extension Method=YaRN2026.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | 50.1 |