Long-context generation on LongBench
23.94QMSumbaseline
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| baselineModel=LLaMA-3.2 3B, #Bits=FP162026.03 | 23.94 | — | — | — | — | 42.98 | — | 64.42 | — | |
| baselineModel=LLaMA-3 8B, #Bits=FP162026.03 | 23.43 | — | — | — | — | 46.26 | — | 66.56 | — | |
| SERQModel=LLaMA-3 8B, #Bits=W4A82026.03 | 22.97 | — | — | — | — | 44.27 | — | 63.34 | — | |
| SERQModel=LLaMA-3.2 3B, #Bits=W4A82026.03 | 22.25 | — | — | — | — | 42.47 | — | 60.78 | — | |
| L2QERModel=LLaMA-3.2 3B, #Bits=W4A82026.03 | 21.53 | — | — | — | — | 40.63 | — | 60.18 | — | |
| FP16Backbone=Vicuna-v1.5-13B, Quantization=FP162024.06 | 21.24 | 24.41 | 26.53 | 68 | 86.81 | 41.97 | 27.57 | 43.08 | 42.45 | |
| DuQuantBackbone=Vicuna-v1.5-7B, Quantization=W4A42024.06 | 21.15 | 19.98 | 25.85 | 64 | 78.91 | 42.24 | 23.15 | 47.66 | 40.37 | |
| FP16Backbone=Vicuna-v1.5-7B, Quantization=FP162024.06 | 21.07 | 23.27 | 26.91 | 66 | 82.59 | 41.06 | 25.53 | 48.23 | 41.83 | |
| DuQuantBackbone=Vicuna-v1.5-13B, Quantization=W4A42024.06 | 20.72 | 18.93 | 26.59 | 66.5 | 83.04 | 42.67 | 26.02 | 38.09 | 40.32 | |
| AtomBackbone=Vicuna-v1.5-7B, Quantization=W4A42024.06 | 20.24 | 17.97 | 24.6 | 58 | 67.2 | 37.94 | 19.41 | 29.34 | 34.34 | |
| AtomBackbone=Vicuna-v1.5-13B, Quantization=W4A42024.06 | 20.23 | 17.67 | 23.39 | 59 | 80.75 | 38.72 | 21.79 | 37.31 | 37.36 | |
| SERQModel=LLaMA-3.2 3B, #Bits=W4A42026.03 | 19.31 | — | — | — | — | 42.14 | — | 53.67 | — | |
| SERQModel=LLaMA-3 8B, #Bits=W4A42026.03 | 19.08 | — | — | — | — | 40.85 | — | 54.78 | — | |
| LLM.int4 ()Model=LLaMA-3.2 3B, #Bits=W4A82026.03 | 15.83 | — | — | — | — | 36.42 | — | 35.24 | — | |
| L2QERModel=LLaMA-3.2 3B, #Bits=W4A42026.03 | 14.92 | — | — | — | — | 33.87 | — | 41.99 | — | |
| LLM.int4 ()Model=LLaMA-3 8B, #Bits=W4A82026.03 | 13.29 | — | — | — | — | 34.2 | — | 44.93 | — | |
| L2QERModel=LLaMA-3 8B, #Bits=W4A82026.03 | 7.28 | — | — | — | — | 25.81 | — | 47.03 | — | |
| OmniQuantBackbone=Vicuna-v1.5-7B, Quantization=W4A42024.06 | 3.93 | 1.62 | 2.64 | 1 | 0.81 | 0.61 | 1.87 | 14.97 | 3.43 | |
| SmoothQuantBackbone=Vicuna-v1.5-13B, Quantization=W4A42024.06 | 2.95 | 2.18 | 3.54 | 1.5 | 1.83 | 0.35 | 6.71 | 11.57 | 3.83 | |
| SmoothQuantBackbone=Vicuna-v1.5-7B, Quantization=W4A42024.06 | 2 | 4.11 | 6.05 | 15 | 1.62 | 1.55 | 4.24 | 25.92 | 7.56 | |
| OmniQuantBackbone=Vicuna-v1.5-13B, Quantization=W4A42024.06 | 1.78 | 0.68 | 2.83 | 9 | 1.13 | 0.45 | 13.83 | 8.46 | 4.77 | |
| L2QERModel=LLaMA-3 8B, #Bits=W4A42026.03 | 0.09 | — | — | — | — | 1.98 | — | 5.65 | — | |
| LLaMA2-7BVariant=Baseline2026.01 | — | — | — | — | — | — | — | — | 45 | |
| LLaMA2-7BVariant=LoRA-Drop, drop-ratio (rho)=0.5, temporal window (k)=32026.01 | — | — | — | — | — | — | — | — | 44.7 | |
| LLaMA3-8BVariant=Baseline2026.01 | — | — | — | — | — | — | — | — | 48.5 | |
| LLaMA3-8BVariant=LoRA-Drop, drop-ratio (rho)=0.5, temporal window (k)=32026.01 | — | — | — | — | — | — | — | — | 48.2 | |
| Qwen2.5-7BVariant=Baseline2026.01 | — | — | — | — | — | — | — | — | 47 | |
| Qwen2.5-7BVariant=LoRA-Drop, drop-ratio (rho)=0.5, temporal window (k)=32026.01 | — | — | — | — | — | — | — | — | 46.7 |