Long-context Understanding on RULER (8K to 256K Contexts)
92.88Performance (8K Context)Llama-3.1-8B-Instruct
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Llama-3.1-8B-InstructBackbone=Llama-3.1-8B-Instruct, Attention Mechanism=Full Attention2026.04 | 92.88 | 92.83 | 89.46 | 70.79 | 80.12 | 72.34 | 83.47 | |
| FluxAttn (FA-TA)Backbone=Llama-3.1-8B-Instruct, Attention Mechanism=FluxAttn (FA-TA)2026.04 | 92.72 | 90.53 | 86.45 | 67.78 | 80.63 | 67.09 | 81.5 | |
| TriangleMixBackbone=Llama-3.1-8B-Instruct, Attention Mechanism=TriangleMix2026.04 | 92.44 | 90.76 | 86.75 | 68 | 78.25 | 64.39 | 80.46 | |
| FluxAttn (FA-XA)Backbone=Llama-3.1-8B-Instruct, Attention Mechanism=FluxAttn (FA-XA)2026.04 | 92.43 | 90.85 | 88.23 | 68.56 | 75.86 | 60.8 | 79.51 | |
| DuoAttentionBackbone=Llama-3.1-8B-Instruct, Attention Mechanism=DuoAttention2026.04 | 91.71 | 86.35 | 85.65 | 62.65 | 62.3 | 38.69 | 70.33 | |
| FluxAttn (FA-SSA)Backbone=Llama-3.1-8B-Instruct, Attention Mechanism=FluxAttn (FA-SSA), Configuration=Variant 22026.04 | 90.11 | 79.39 | 79.22 | 56.08 | 62.94 | 59.39 | 73.67 | |
| Qwen3-8BBackbone=Qwen3-8B, Attention Mechanism=Full Attention2026.04 | 89.69 | 85.62 | 63.23 | 82.39 | 65.84 | 66.71 | 75.74 | |
| Qwen3-4BBackbone=Qwen3-4B, Attention Mechanism=Full Attention2026.04 | 87.49 | 86.82 | 60.05 | 70.98 | 53.19 | 43.27 | 66 | |
| FluxAttn (FA-TA)Backbone=Qwen3-8B, Attention Mechanism=FluxAttn (FA-TA)2026.04 | 87.49 | 86.17 | 60.85 | — | 60.75 | 63.03 | 73.51 | |
| TriangleMixBackbone=Qwen3-4B, Attention Mechanism=TriangleMix2026.04 | 87.42 | 85.1 | 58.73 | 67.94 | 50.97 | 44.47 | 63.74 | |
| PruLongBackbone=Llama-3.1-8B-Instruct, Attention Mechanism=PruLong2026.04 | 86.96 | 76.55 | 70.65 | 54.52 | 48.18 | 30 | 59.87 | |
| FluxAttn (FA-XA)Backbone=Qwen3-4B, Attention Mechanism=FluxAttn (FA-XA)2026.04 | 86.79 | 84.94 | 59.52 | 68.82 | 51.77 | 43.43 | 63.67 | |
| DuoAttentionBackbone=Qwen3-8B, Attention Mechanism=DuoAttention2026.04 | 86.68 | 86.01 | 63.23 | 77.52 | 61.5 | 61.95 | 72.41 | |
| FluxAttn (FA-XA)Backbone=Qwen3-8B, Attention Mechanism=FluxAttn (FA-XA)2026.04 | 85.88 | 85.54 | 65.08 | 81.95 | 65.09 | 65.38 | 74.65 | |
| FluxAttn (FA-TA)Backbone=Qwen3-4B, Attention Mechanism=FluxAttn (FA-TA)2026.04 | 84.28 | 84.53 | 60.58 | 68.6 | 51.91 | 51.64 | 65.55 | |
| FluxAttn (FA-SSA)Backbone=Qwen3-8B, Attention Mechanism=FluxAttn (FA-SSA)2026.04 | 84.09 | 81.9 | 60.58 | 79.3 | 64.74 | 65.27 | 73.03 | |
| PruLongBackbone=Qwen3-8B, Attention Mechanism=PruLong2026.04 | 83.85 | 80.86 | 60.05 | 77.25 | 62.54 | 61.49 | 70.97 | |
| FluxAttn (FA-SSA)Backbone=Qwen3-8B, Attention Mechanism=FluxAttn (FA-SSA), Configuration=Variant 22026.04 | 83.54 | 81 | 59.79 | 77.93 | 64.83 | 65.12 | 72.51 | |
| FluxAttn (FA-SSA)Backbone=Llama-3.1-8B-Instruct, Attention Mechanism=FluxAttn (FA-SSA)2026.04 | 82.88 | 78.09 | 70.39 | 52.29 | 62.2 | 50.73 | 76.75 | |
| FluxAttn (FA-SSA)Backbone=Qwen3-4B, Attention Mechanism=FluxAttn (FA-SSA)2026.04 | 81.58 | 82.11 | 58.73 | 72.89 | 52.81 | 56.91 | 66.95 | |
| TriangleMixBackbone=Qwen3-8B, Attention Mechanism=TriangleMix2026.04 | 81.01 | 75.67 | 63.49 | 73.76 | 61.54 | 66.84 | 70.47 | |
| FluxAttn (FA-SSA)Backbone=Qwen3-4B, Attention Mechanism=FluxAttn (FA-SSA), Configuration=Variant 22026.04 | 80.36 | 80.75 | 56.08 | 71.49 | 59.17 | 56 | 67.19 | |
| DuoAttentionBackbone=Qwen3-4B, Attention Mechanism=DuoAttention2026.04 | 79.38 | 76.08 | 52.91 | 69.02 | 43.28 | 44.96 | 60.67 | |
| PruLongBackbone=Qwen3-4B, Attention Mechanism=PruLong2026.04 | 74.21 | 75.72 | 47.88 | 59.27 | 47.1 | 45.69 | 60.25 |