Long-context Language Understanding on LongBench (Full Metric Set)
58.4Average ScoreOriginal*
Evaluation results
| Method | Links | |||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Original*SWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 58.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Original*SWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 57.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 56.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FISHERSWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 55.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Original*SWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 54.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FullBackbone=LLaMA-3.1-70B-Instruct2026.06 | 54.62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RaBitQCacheBackbone=LLaMA-3.1-70B-Instruct2026.06 | 54.58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SparQBackbone=LLaMA-3.1-70B-Instruct2026.06 | 53.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| APLSWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 53.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full-contextCompression Strategy=Full-context, Prefill (%)=100%, KV (%)=100%2026.04 | 52.73 | 50.16 | — | 36.54 | 63.41 | 63.85 | — | 29.41 | 56.53 | — | 34.82 | 31.85 | 12.53 | — | — | 48.22 | 27.83 | 70.57 | 49.83 | 76.55 | 91.52 | — | 100 | |
| DCAMSWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 52.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| StructKVCompression Strategy=Prefill-Aware, Prefill (%)=60%, KV (%)=20%2026.04 | 52.44 | 49.93 | — | 36.25 | 62.85 | 63.15 | — | 29.24 | 56.12 | — | 34.46 | 31.75 | 12.43 | — | — | 48.13 | 27.76 | 69.82 | 49.66 | 76.13 | 91.35 | — | 100 | |
| BOSCH-layerSWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 52.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RAZORSWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 52.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 52.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| StructKVCompression Strategy=Prefill-Aware, Prefill (%)=60%, KV (%)=10%2026.04 | 51.87 | 49.82 | — | 36.15 | 61.54 | 62.76 | — | 28.53 | 55.23 | — | 33.26 | 31.62 | 12.16 | — | — | 48.14 | 27.24 | 66.84 | 49.54 | 75.82 | 91.26 | — | 100 | |
| FastKVCompression Strategy=Prefill-Aware, Prefill (%)=60%, KV (%)=20%2026.04 | 51.12 | 48.92 | — | 35.92 | 59.24 | 62.25 | — | 28.14 | 54.43 | — | 32.55 | 31.22 | 11.53 | — | — | 47.86 | 26.83 | 64.12 | 49.12 | 75.52 | 90.84 | — | 99.5 | |
| APLSWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 51.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCH-multiSWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 51.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVCompression Strategy=Decoding-Only, Prefill (%)=100%, KV (%)=20%2026.04 | 51 | 49.63 | — | 35.84 | 59.86 | 61.88 | — | 28.23 | 54.52 | — | 32.45 | 30.93 | 11.85 | — | — | 47.84 | 26.55 | 61.26 | 49.54 | 75.02 | 91.06 | — | 99.5 | |
| QuestBackbone=LLaMA-3.1-70B-Instruct2026.06 | 50.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FastKVCompression Strategy=Prefill-Aware, Prefill (%)=60%, KV (%)=10%2026.04 | 50.53 | 48.21 | — | 35.52 | 58.54 | 61.53 | — | 27.63 | 53.92 | — | 31.86 | 30.84 | 11.08 | — | — | 47.53 | 26.24 | 63.26 | 48.54 | 74.05 | 90.26 | — | 99.5 | |
| PROXYSWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 50.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SnapKVCompression Strategy=Decoding-Only, Prefill (%)=100%, KV (%)=10%2026.04 | 50.43 | 49.13 | — | 35.42 | 59.25 | 61.55 | — | 27.93 | 53.86 | — | 31.54 | 30.54 | 11.23 | — | — | 47.52 | 25.86 | 60.58 | 49.25 | 73.55 | 90.24 | — | 99.5 | |
| BOSCH-singleSWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 50.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PROXYSWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 49.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| StreamingLLMCompression Strategy=Decoding-Only, Prefill (%)=100%, KV (%)=20%2026.04 | 49.41 | 48.85 | — | 35.82 | 55.63 | 62.18 | — | 27.91 | 52.86 | — | 29.54 | 30.12 | 10.25 | — | — | 47.14 | 25.93 | 61.24 | 48.96 | 75.04 | 91.02 | — | 88 | |
| StreamingLLMCompression Strategy=Decoding-Only, Prefill (%)=100%, KV (%)=10%2026.04 | 48.68 | 48.33 | — | 35.12 | 54.21 | 61.25 | — | 27.44 | 52.14 | — | 28.45 | 28.53 | 9.54 | — | — | 46.82 | 25.26 | 60.56 | 48.63 | 74.05 | 90.52 | — | 88 | |
| RANDSWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 48.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GemFilterCompression Strategy=Prefill-Aware, Prefill (%)=70%, KV (%)=20%2026.04 | 47.87 | 47.85 | — | 34.63 | 53.52 | 60.52 | — | 26.85 | 48.94 | — | 31.24 | 27.15 | 8.24 | — | — | 46.83 | 25.42 | 50.23 | 46.54 | 74.55 | 90.83 | — | 92.5 | |
| BOSCH-layerSWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 47.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.5, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GemFilterCompression Strategy=Prefill-Aware, Prefill (%)=60%, KV (%)=10%2026.04 | 46.84 | 46.94 | — | 33.92 | 52.86 | 59.86 | — | 26.54 | 47.55 | — | 29.53 | 26.24 | 7.55 | — | — | 46.12 | 24.63 | 48.52 | 45.83 | 73.05 | 90.22 | — | 90 | |
| BOSCH-multiSWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 46.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Diagonal-Tiled Mixed-Precision AttentionBackbone=LLaMA3.1-8B2026.04 | 46.43 | 32.54 | 32.93 | 34.89 | 50.49 | 56.96 | 44.5 | 27.07 | 52.65 | 58.81 | 26.81 | 26.7 | 6.22 | 96 | 85.5 | 43.05 | 24.67 | 54 | 43.99 | 71.5 | 87.89 | 17.92 | — | |
| BOSCHSWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 46.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCH-singleSWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 45.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Original*SWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 45.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PyramidInferCompression Strategy=Prefill-Aware, Prefill (%)=60%, KV (%)=60%2026.04 | 44.78 | 44.53 | — | 29.54 | 45.16 | 60.84 | — | 21.86 | 43.25 | — | 24.25 | 27.54 | 5.83 | — | — | 39.82 | 23.42 | 61.52 | 47.56 | 70.05 | 89.24 | — | 82 | |
| FISHERSWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 44.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Native (SDPA)Backbone=LLaMA3.1-8B2026.04 | 44.11 | 39.15 | 28.4 | 33.87 | 49.89 | 49.46 | 41.5 | 26.16 | 52.26 | 57.57 | 25.82 | 25.28 | 4.02 | 99 | 92.14 | 42.64 | 23.47 | 42.23 | 36.6 | 65 | 77.01 | 14.8 | — | |
| QADASWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 43.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| INTRSWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 43.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| INTRSWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 43.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DCAMSWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 43 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| INTRSWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 42.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| APLSWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 41.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.5, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 41.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RAZORSWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 41.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FISHERSWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 40.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DCAMSWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 40.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCH-multiSWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 40.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| H2OCompression Strategy=Decoding-Only, Prefill (%)=100%, KV (%)=20%2026.04 | 40.73 | 23.83 | — | 34.92 | 24.12 | 62.54 | — | 27.56 | 32.85 | — | 18.54 | 26.84 | 9.06 | — | — | 19.56 | 24.23 | 58.26 | 47.12 | 72.04 | 92.15 | — | 78 | |
| RANDSWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 40.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RANDSWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 40.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QADASWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCH-layerSWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 39.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| H2OCompression Strategy=Decoding-Only, Prefill (%)=100%, KV (%)=10%2026.04 | 39.11 | 19.54 | — | 35.22 | 20.87 | 59.83 | — | 28.14 | 30.55 | — | 13.26 | 25.42 | 8.54 | — | — | 16.23 | 25.65 | 57.56 | 46.53 | 71.55 | 91.82 | — | 75 | |
| PROXYSWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 38.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.75, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Diagonal-Tiled Mixed-Precision AttentionBackbone=LLaMA3.2-3B2026.04 | 37.2 | 36.33 | 33.04 | 33.04 | 48.59 | 45.54 | 30.75 | 21.78 | 47.87 | 53.62 | 25.77 | 25.44 | 1.5 | 37 | 10.5 | 39.62 | 23.63 | 50.15 | 42.74 | 69.5 | 88.13 | 16.69 | — | |
| BOSCH-layerSWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 37.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| APLSWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 36.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QADASWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 36.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCH-singleSWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 36.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FISHERSWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 36.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.875, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 36.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| INTRSWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCH-multiSWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.5, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Native (SDPA)Backbone=LLaMA3.2-3B2026.04 | 35.84 | 30.75 | 28.11 | 32.26 | 46.18 | 42.98 | 26.75 | 25.56 | 46.84 | 48.06 | 20.35 | 19.76 | 3.4 | 80 | 8.25 | 32.13 | 22.52 | 44.14 | 34.79 | 62.5 | 83.68 | 13.65 | — | |
| QADASWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 35.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RAZORSWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 35.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RAZORSWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 34.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PROXYSWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 34.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCH-singleSWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RANDSWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 33.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BMESWA rho ratio=0.25, Model Size=14B, Evaluation Mode=Zero-shot2026.04 | 32.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DCAMSWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 32.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.5, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 32.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.75, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 31.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BMESWA rho ratio=0.25, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 31.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BMESWA rho ratio=0.25, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.875, Model Size=8B, Evaluation Mode=Zero-shot2026.04 | 28.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BMESWA rho ratio=0.25, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 28.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.75, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 26 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.75, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 24.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.875, Model Size=1.7B, Evaluation Mode=Zero-shot2026.04 | 23.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOSCHSWA rho ratio=0.875, Model Size=30B, Evaluation Mode=Zero-shot2026.04 | 19.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |