Long-context language understanding on LongBench 16 English tasks
43.5Accuracy (SingQA)FP16
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| FP16Backbone=Llama-3.1-8B-Instruct, Bits=16, residual bit-width=b=2, chunk size=128, block size=128x1282026.04 | 43.5 | 44.9 | 29.2 | 69.4 | 53.9 | 53.1 | 49 | |
| eOptShrinkQprodBackbone=Llama-3.1-8B-Instruct, Bits=3.22, residual bit-width=b=2, chunk size=128, block size=128x128, SVD quantization=4 bits2026.04 | 43.1 | 42.6 | 28.5 | 67.9 | 50.5 | 53.1 | 47.6 | |
| FP16Backbone=Ministral-8B-Instruct, Bits=16, residual bit-width=b=2, chunk size=128, block size=128x1282026.04 | 42.5 | 49.5 | 27.8 | 71 | 54.5 | 57.3 | 50.5 | |
| eOptShrinkQMSEBackbone=Llama-3.1-8B-Instruct, Bits=2.22, residual bit-width=b=2, chunk size=128, block size=128x128, SVD quantization=4 bits2026.04 | 42.3 | 42.7 | 27.4 | 67.4 | 50.5 | 53.9 | 47.4 | |
| TQprodBackbone=Llama-3.1-8B-Instruct, Bits=3.00, residual bit-width=b=2, chunk size=128, block size=128x1282026.04 | 41.9 | 38.6 | 27.6 | 67.5 | 47.2 | 48.5 | 45.2 | |
| SVDr=1+TQBackbone=Llama-3.1-8B-Instruct, Bits=2.04, residual bit-width=b=2, chunk size=128, block size=128x128, SVD quantization=4 bits2026.04 | 41.5 | 39.4 | 26.3 | 66.9 | 49.4 | 52.2 | 46 | |
| TQMSEBackbone=Llama-3.1-8B-Instruct, Bits=2.00, residual bit-width=b=2, chunk size=128, block size=128x1282026.04 | 41 | 37.9 | 27.1 | 67.8 | 46.4 | 49.2 | 44.9 | |
| eOptShrinkQprodBackbone=Ministral-8B-Instruct, Bits=3.22, residual bit-width=b=2, chunk size=128, block size=128x128, SVD quantization=4 bits2026.04 | 40.9 | 46.3 | 27.1 | 69.4 | 50.8 | 57 | 48.6 | |
| eOptShrinkQMSEBackbone=Ministral-8B-Instruct, Bits=2.22, residual bit-width=b=2, chunk size=128, block size=128x128, SVD quantization=4 bits2026.04 | 40.1 | 46.6 | 26.3 | 69.2 | 50.5 | 56.9 | 48.3 | |
| SVDr=1+TQBackbone=Ministral-8B-Instruct, Bits=2.04, residual bit-width=b=2, chunk size=128, block size=128x128, SVD quantization=4 bits2026.04 | 39.8 | 42.3 | 25.5 | 68.4 | 47.5 | 55.7 | 46.5 | |
| TQprodBackbone=Ministral-8B-Instruct, Bits=3.00, residual bit-width=b=2, chunk size=128, block size=128x1282026.04 | 39.5 | 42.6 | 26.7 | 68.5 | 47.8 | 54.5 | 46.6 | |
| TQMSEBackbone=Ministral-8B-Instruct, Bits=2.00, residual bit-width=b=2, chunk size=128, block size=128x1282026.04 | 39.3 | 42.6 | 26.2 | 68.4 | 48.2 | 55 | 46.6 |