Long-context Language Understanding on LongBench (Specific Metric Subset including NQA and Delta Avg)
34.92NQAH2O
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| H2OBackbone=Llama-3.1-70B2026.04 | 34.92 | 54.31 | 64.45 | 47.08 | 32.53 | 34.92 | 98 | 69.73 | 54.49 | -0.15 | |
| Full AttentionBackbone=Llama-3.1-70B2026.04 | 34.81 | 54.24 | 64.45 | 47.08 | 32.7 | 34.93 | 98 | 70.88 | 54.64 | — | |
| SinkRouterBackbone=Llama-3.1-70B2026.04 | 34.43 | 54.67 | 64.35 | 46.75 | 33.01 | 34.84 | 98 | 70.46 | 54.56 | -0.08 | |
| FastGenBackbone=Llama-3.1-70B2026.04 | 30.24 | 52.32 | 62.96 | 41.28 | 19.62 | 31.62 | 95.29 | 68.01 | 50.17 | -4.47 | |
| SinkRouterBackbone=Llama-3.1-8B2026.04 | 30.08 | 53.96 | 55.71 | 29.62 | 32.14 | 33.87 | 99.5 | 64.24 | 49.89 | -0.69 | |
| Full AttentionBackbone=Llama-3.1-8B2026.04 | 30 | 55.14 | 56.15 | 30.08 | 33.85 | 35.2 | 99.5 | 64.73 | 50.58 | — | |
| H2OBackbone=Llama-3.1-8B2026.04 | 29.69 | 54.99 | 56.1 | 30.55 | 34.11 | 35.15 | 99.5 | 64.81 | 50.61 | 0.03 | |
| FastGenBackbone=Llama-3.1-8B2026.04 | 26.02 | 52.92 | 52.79 | 26.54 | 18.75 | 31.61 | 92.5 | 64.21 | 45.67 | -4.91 | |
| StreamingLLMBackbone=Llama-3.1-70B2026.04 | 25.35 | 46.42 | 55.23 | 26.15 | 16.39 | 33.41 | 60.5 | 68.95 | 41.55 | -13.09 | |
| StreamingLLMBackbone=Llama-3.1-8B2026.04 | 20.7 | 38.05 | 40.18 | 15.9 | 14.93 | 31.29 | 36.5 | 65.03 | 32.82 | -17.76 | |
| H2OBackbone=Yi-9B-200K2026.04 | 12.25 | 36.56 | 50.73 | 26.95 | 21.04 | 29.3 | 58 | 72.84 | 38.46 | -0.81 | |
| Full AttentionBackbone=Yi-9B-200K2026.04 | 11.92 | 37.6 | 51.66 | 27.55 | 21.66 | 30.62 | 59.5 | 73.62 | 39.27 | — | |
| SinkRouterBackbone=Yi-9B-200K2026.04 | 11.66 | 37.24 | 52.01 | 27.81 | 20.81 | 30.13 | 50.5 | 73.22 | 37.92 | -1.35 | |
| FastGenBackbone=Yi-9B-200K2026.04 | 10.65 | 34.29 | 43.97 | 17.52 | 12.94 | 24.43 | 33.5 | 71.87 | 31.15 | -8.12 | |
| StreamingLLMBackbone=Yi-9B-200K2026.04 | 4.29 | 31.17 | 43.07 | 17.93 | 13.75 | 22.61 | 19.5 | 62.83 | 26.89 | -12.38 |