Long-context language understanding on LongBench (SQA, MQA, Sum, FS, Syn, Code, Avg)
50.91SQA ScoreMISA†
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| MISA†Model=DeepSeek-V3.2, Indexer=MISA†, Heads=82026.05 | 50.91 | 52.27 | 22.04 | 62.1 | 69.83 | 48.45 | 50.95 | |
| DSAModel=DeepSeek-V3.2, Indexer=DSA, Heads=642026.05 | 50.89 | 52.66 | 22.11 | 62.24 | 69.83 | 48.56 | 51.05 | |
| MISAModel=DeepSeek-V3.2, Indexer=MISA, Heads=82026.05 | 50.83 | 51.62 | 22.31 | 61.77 | 70 | 48.54 | 50.85 | |
| HISAModel=DeepSeek-V3.2, Indexer=HISA, Heads=642026.05 | 49.17 | 51.96 | 22.13 | 61.62 | 70.83 | 48.99 | 50.78 | |
| BlockModel=DeepSeek-V3.2, Indexer=Block, Heads=642026.05 | 48.36 | 49.76 | 21.9 | 59.45 | 68.67 | 49.09 | 49.54 | |
| HISAModel=GLM-5, Indexer=HISA, Heads=322026.05 | 42.45 | 27.62 | 17.9 | 63.78 | 69.35 | 56.79 | 46.32 | |
| MISAModel=GLM-5, Indexer=MISA, Heads=82026.05 | 41.64 | 29.16 | 18.22 | 63.53 | 68.76 | 57.24 | 46.43 | |
| DSAModel=GLM-5, Indexer=DSA, Heads=322026.05 | 41.23 | 27.89 | 18.39 | 63.2 | 68.84 | 56.53 | 46.01 | |
| MISA†Model=GLM-5, Indexer=MISA†, Heads=82026.05 | 41.16 | 28.67 | 18.56 | 63.57 | 69.26 | 57.83 | 46.51 | |
| BlockModel=GLM-5, Indexer=Block, Heads=322026.05 | 38.35 | 24.29 | 16.95 | 60.64 | 60.49 | 55.29 | 42.67 | |
| Llama-3.1-8B-Instruct (BF16)Precision=BF16, Token Density=100% (Dense)2025.10 | 22.9 | 16.9 | 28.6 | 71 | 49.4 | 60.3 | 41.5 | |
| DHSABackbone=Llama-3.1-8B-Instruct (BF16), Token Density=12.5%, Sparsity Setting=Matched2025.10 | 19.1 | 16.2 | 26.1 | 71.5 | 51.3 | 61.4 | 40.4 | |
| BlockSparseBackbone=Llama-3.1-8B-Instruct (BF16), Token Density=12.5%, Sparsity Setting=Matched2025.10 | 15.3 | 8.8 | 21.6 | 64.9 | 40.4 | 49.5 | 33.4 | |
| Qwen2.5-14B-Instruct (4-bit)Precision=4-bit, Token Density=100% (Dense)2025.10 | 14.3 | 9.9 | 22.8 | 69.8 | 48.6 | 53.5 | 36.5 | |
| DHSABackbone=Qwen2.5-14B-Instruct (4-bit), Token Density=12.5%, Sparsity Setting=Matched2025.10 | 12.5 | 10.2 | 21.7 | 70.1 | 43 | 48.1 | 34.3 | |
| BlockSparseBackbone=Qwen2.5-14B-Instruct (4-bit), Token Density=12.5%, Sparsity Setting=Matched2025.10 | 9.9 | 6.4 | 21.7 | 62 | 11.3 | 30.6 | 23.6 |