Long-context understanding on LongBench (eval)
39.6Qasper ScoreFixed Sparse (16, 0.02)
Evaluation results
| Method | Links | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Fixed Sparse (16, 0.02)Model Backbone=Qwen2.5-7B-Instruct, blk=16, bgt=0.022026.05 | 39.6 | 52.95 | 55.58 | 29.26 | 23.42 | 87.39 | 44 | 9 | 100 | 60.08 | 27.96 | 32.54 | 23.47 | 16.11 | 42.95 | — | |
| Fluxion (Qwen2.5-7B)Model Backbone=Qwen2.5-7B-Instruct, Sparsity Method=Fluxion2026.05 | 39.6 | 55.54 | 56.36 | 29.05 | 22.91 | 87.24 | 42.62 | 9 | 100 | 59.46 | 29.78 | 33.6 | 23.18 | 16.66 | 43.21 | -0.13 | |
| Fixed Sparse (32, 0.05)Model Backbone=Llama-3.1-8B-Instruct, blk=32, bgt=0.052026.05 | 39.43 | 52.56 | 60.13 | 34.45 | 24.75 | 94.1 | 44.9 | 9.33 | 100 | 65.52 | 31.69 | 31.54 | 25.21 | 16.21 | 44.99 | — | |
| Fixed Sparse (16, 0.05)Model Backbone=Qwen2.5-7B-Instruct, blk=16, bgt=0.052026.05 | 38.74 | 53.31 | 56.98 | 29.67 | 23.41 | 89.03 | 43.06 | 9 | 100 | 60.68 | 28.87 | 32.03 | 24.2 | 16.33 | 43.24 | — | |
| Fixed Sparse (32, 0.05)Model Backbone=Qwen2.5-7B-Instruct, blk=32, bgt=0.052026.05 | 38.64 | 51.31 | 56.36 | 29.18 | 22.95 | 88.1 | 43.9 | 9 | 100 | 60.8 | 28.79 | 31.89 | 23.47 | 16.29 | 42.91 | — | |
| Qwen2.5-7B-Instruct (FULL)Model Backbone=Qwen2.5-7B-Instruct, Sparsity Method=FULL2026.05 | 38.63 | 51.21 | 58.23 | 29.42 | 23.3 | 89.01 | 43.48 | 9 | 100 | 61.49 | 28.22 | 32.95 | 24.64 | 17.19 | 43.34 | — | |
| Fixed Sparse (16, 0.02)Model Backbone=Llama-3.1-8B-Instruct, blk=16, bgt=0.022026.05 | 38.49 | 57.06 | 60.13 | 33.97 | 23.87 | 94.1 | 43.33 | 10 | 99 | 65.15 | 30.76 | 32.1 | 24.73 | 16.02 | 44.91 | — | |
| Fixed Sparse (32, 0.02)Model Backbone=Qwen2.5-7B-Instruct, blk=32, bgt=0.022026.05 | 38.19 | 51.32 | 53.83 | 28.23 | 22.89 | 88.33 | 42.93 | 9 | 100 | 57.01 | 27.31 | 30.08 | 23.42 | 15.41 | 42 | — | |
| Fixed Sparse (16, 0.05)Model Backbone=Llama-3.1-8B-Instruct, blk=16, bgt=0.052026.05 | 37.67 | 55.15 | 60.13 | 33.95 | 24.43 | 94.1 | 44.05 | 9 | 100 | 64.84 | 31.21 | 31.7 | 25.01 | 16.22 | 44.82 | — | |
| Fluxion (Llama-3.1-8B)Model Backbone=Llama-3.1-8B-Instruct, Sparsity Method=Fluxion2026.05 | 37.64 | 56.93 | 60.13 | 34.2 | 24.43 | 94.1 | 43.63 | 9 | 100 | 64.95 | 31.27 | 32.89 | 24.81 | 16.6 | 45.04 | 0.02 | |
| Llama-3.1-8B-Instruct (FULL)Model Backbone=Llama-3.1-8B-Instruct, Sparsity Method=FULL2026.05 | 37.19 | 54.54 | 60.13 | 34.25 | 24.18 | 94.1 | 43.91 | 9 | 100 | 65.3 | 31.97 | 33.55 | 25 | 17.12 | 45.02 | — | |
| Fixed Sparse (32, 0.02)Model Backbone=Llama-3.1-8B-Instruct, blk=32, bgt=0.022026.05 | 36.37 | 54.01 | 60.27 | 33.95 | 23.46 | 94.1 | 44.35 | 10.06 | 99 | 66.49 | 30.94 | 30.51 | 24.31 | 15.54 | 44.53 | — |