Language Modeling on WikiText-2 (val)
4.73Perplexity (BVS)FP16
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FP16Backbone=MISTRAL-7B, BIT=162026.07 | 4.73 | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=MISTRAL-7B, BIT=22026.07 | 4.98 | — | — | — | — | — | — | — | — | — | — | |
| ANTKVBackbone=MISTRAL-7B, BIT=22026.07 | 5.08 | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=MISTRAL-7B, BIT=22026.07 | 5.11 | — | — | — | — | — | — | — | — | — | — | |
| FP16Backbone=LLAMA-2-7B, BIT=162026.07 | 5.12 | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=LLAMA-2-7B, BIT=22026.07 | 5.34 | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=LLAMA-2-7B, BIT=22026.07 | 5.42 | — | — | — | — | — | — | — | — | — | — | |
| ANTKVBackbone=LLAMA-2-7B, BIT=22026.07 | 5.51 | — | — | — | — | — | — | — | — | — | — | |
| FP16Backbone=LLAMA-3-8B, BIT=162026.07 | 5.54 | — | — | — | — | — | — | — | — | — | — | |
| BF16Backbone=Llama-3, Parameters=8B, Precision=BF162026.06 | 5.89 | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=LLAMA-3-8B, BIT=22026.07 | 5.91 | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=LLAMA-3-8B, BIT=22026.07 | 6.09 | — | — | — | — | — | — | — | — | — | — | |
| ANTKVBackbone=LLAMA-3-8B, BIT=22026.07 | 6.1 | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=LLAMA-2-7B, BIT=12026.07 | 6.11 | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=MISTRAL-7B, BIT=12026.07 | 6.11 | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=LLAMA-2-7B, BIT=0.752026.07 | 6.81 | — | — | — | — | — | — | — | — | — | — | |
| BF16Backbone=Qwen-2.5, Parameters=7B, Precision=BF162026.06 | 6.92 | — | — | — | — | — | — | — | — | — | — | |
| BF16Backbone=Llama-3, Parameters=3B, Precision=BF162026.06 | 7.14 | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=MISTRAL-7B, BIT=0.752026.07 | 7.14 | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=LLAMA-3-8B, BIT=12026.07 | 7.2 | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=MISTRAL-7B, BIT=12026.07 | 7.25 | — | — | — | — | — | — | — | — | — | — | |
| ANTKVBackbone=MISTRAL-7B, BIT=12026.07 | 7.32 | — | — | — | — | — | — | — | — | — | — | |
| ANTKVBackbone=MISTRAL-7B, BIT=0.752026.07 | 7.41 | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=MISTRAL-7B, BIT=0.752026.07 | 7.64 | — | — | — | — | — | — | — | — | — | — | |
| FP16 BaseBackbone=LLaMA3-3B, Weight bit-width (W)=16, Activation bit-width (A)=16, Quantization Protocol=None (Full Precision)2026.05 | 7.7 | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=LLAMA-2-7B, BIT=12026.07 | 7.75 | — | — | — | — | — | — | — | — | — | — | |
| BF16Backbone=Qwen-2.5, Parameters=3B, Precision=BF162026.06 | 7.88 | — | — | — | — | — | — | — | — | — | — | |
| ANTKVBackbone=LLAMA-2-7B, BIT=12026.07 | 7.92 | — | — | — | — | — | — | — | — | — | — | |
| ANTKVBackbone=LLAMA-2-7B, BIT=0.752026.07 | 8.21 | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=LLAMA-2-7B, BIT=0.752026.07 | 8.39 | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=LLAMA-3-8B, BIT=0.752026.07 | 8.46 | — | — | — | — | — | — | — | — | — | — | |
| Llama-3.1-8B-Instruct2026.02 | 8.52 | — | — | — | 891.84 | 892.89 | 1,148.18 | — | — | — | — | |
| Qwen2.5-7B-Instruct2026.02 | 8.87 | — | — | — | 866.06 | 867.71 | 1,182.36 | — | — | — | — | |
| BF16Backbone=Qwen-2.5, Parameters=1.5B, Precision=BF162026.06 | 8.89 | — | — | — | — | — | — | — | — | — | — | |
| BF16Backbone=Llama-3, Parameters=1B, Precision=BF162026.06 | 8.97 | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=LLAMA-2-7B, BIT=0.3752026.07 | 9.42 | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=LLAMA-3-8B, BIT=12026.07 | 9.56 | — | — | — | — | — | — | — | — | — | — | |
| FP16 BaselineBackbone=LLaMA3-1B, Weight precision=FP16, Activation precision=FP162026.05 | 9.6 | — | — | — | — | — | — | — | — | — | — | |
| ANTKVBackbone=LLAMA-3-8B, BIT=12026.07 | 9.62 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-3B-Instruct2026.02 | 9.91 | — | — | — | 397.85 | 401.55 | 2,573.84 | — | — | — | — | |
| GSRQBackbone=MISTRAL-7B, BIT=0.3752026.07 | 10.07 | — | — | — | — | — | — | — | — | — | — | |
| SmolLM3-3B2026.02 | 10.2 | — | — | — | 392.89 | 393.52 | 2,606.35 | — | — | — | — | |
| SmolLM2-1.7B-Instruct2026.02 | 10.33 | — | — | — | 240.48 | 240.89 | 4,258.23 | — | — | — | — | |
| ANTKVBackbone=LLAMA-3-8B, BIT=0.752026.07 | 10.41 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5BVariant=Baseline2026.02 | 10.79 | — | — | — | 216.99 | 217.13 | 4,719.07 | — | — | — | — | |
| CQBackbone=LLAMA-3-8B, BIT=0.752026.07 | 11.18 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5BModification=SR, Replaced Layers=(15)2026.02 | 11.24 | — | — | — | 211.87 | 212.07 | 4,833.25 | — | — | — | — | |
| ANTKVBackbone=MISTRAL-7B, BIT=0.3752026.07 | 11.65 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5BModification=SR, Replaced Layers=(15, 17)2026.02 | 11.77 | — | — | — | 207.14 | 207.58 | 4,943.49 | — | — | — | — | |
| WinQ + JacQuantBackbone=LLaMA3-1B, Weight precision=W2, Activation precision=A162026.05 | 11.8 | — | — | — | — | — | — | — | — | — | — | |
| WinQBackbone=LLaMA3-1B, Weight precision=W2, Activation precision=A162026.05 | 11.9 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQ + JacQuantBackbone=LLaMA3-1B, Weight precision=W2, Activation precision=A162026.05 | 12.3 | — | — | — | — | — | — | — | — | — | — | |
| WinQ + JacQuantBackbone=LLaMA3-1B, Weight precision=W1.58, Activation precision=A162026.05 | 12.3 | — | — | — | — | — | — | — | — | — | — | |
| Llama-3.2-3B-Instruct2026.02 | 12.32 | — | — | — | 414.51 | 416.03 | 2,470.36 | — | — | — | — | |
| Qwen2.5-1.5BModification=SR, Replaced Layers=(15, 17, 10)2026.02 | 12.44 | — | — | — | 202.16 | 202.41 | 5,065.23 | — | — | — | — | |
| ParetoQ + JacQuantBackbone=LLaMA3-3B, Weight bit-width (W)=1.58, Activation bit-width (A)=8, Quantization Protocol=QAT2026.05 | 12.7 | — | — | — | — | — | — | — | — | — | — | |
| GSRQBackbone=LLAMA-3-8B, BIT=0.3752026.07 | 12.77 | — | — | — | — | — | — | — | — | — | — | |
| WinQBackbone=LLaMA3-1B, Weight precision=W1.58, Activation precision=A162026.05 | 12.9 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQBackbone=LLaMA3-3B, Weight bit-width (W)=1.58, Activation bit-width (A)=8, Quantization Protocol=QAT2026.05 | 13.1 | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=MISTRAL-7B, BIT=0.3752026.07 | 13.2 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQ + Rotation TrickBackbone=LLaMA3-1B, Weight precision=W2, Activation precision=A162026.05 | 13.3 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5BModification=SR, Replaced Layers=(15, 17, 10, 13)2026.02 | 13.34 | — | — | — | 197.64 | 197.86 | 5,181.18 | — | — | — | — | |
| ANTKVBackbone=LLAMA-2-7B, BIT=0.3752026.07 | 13.37 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQBackbone=LLaMA3-1B, Weight precision=W2, Activation precision=A162026.05 | 13.5 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQ + JacQuantBackbone=LLaMA3-1B, Weight precision=W1.58, Activation precision=A162026.05 | 13.9 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQ + Rotation TrickBackbone=LLaMA3-1B, Weight precision=W1.58, Activation precision=A162026.05 | 14.2 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5BModification=SR, Replaced Layers=(15, 17, 10, 13, 14)2026.02 | 14.21 | — | — | — | 193.12 | 193.32 | 5,302.48 | — | — | — | — | |
| Llama-3.2-1B-Instruct2026.02 | 14.21 | — | — | — | 169.31 | 170.08 | 6,048.19 | — | — | — | — | |
| TinyLlama-1.1B-Chat-v1.02026.02 | 14.8 | — | — | — | 149 | 149.44 | 6,872.68 | — | — | — | — | |
| ParetoQBackbone=LLaMA3-1B, Weight precision=W1.58, Activation precision=A162026.05 | 14.8 | — | — | — | — | — | — | — | — | — | — | |
| CQBackbone=LLAMA-2-7B, BIT=0.3752026.07 | 14.82 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5BModification=SR, Replaced Layers=(15, 17, 10, 13, 14, 16)2026.02 | 15.05 | — | — | — | 188.16 | 188.57 | 5,442.06 | — | — | — | — | |
| WinQ + JacQuantBackbone=LLaMA3-1B, Weight precision=W1, Activation precision=A162026.05 | 15.1 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQ + JacQuantBackbone=LLaMA3-3B, Weight bit-width (W)=1, Activation bit-width (A)=8, Quantization Protocol=QAT2026.05 | 15.2 | — | — | — | — | — | — | — | — | — | — | |
| WinQBackbone=LLaMA3-1B, Weight precision=W1, Activation precision=A162026.05 | 15.3 | — | — | — | — | — | — | — | — | — | — | |
| GPTQ + PiSOBackbone=Qwen-2.5, Parameters=7B, Precision=2-bit weight-only, Strategy=Decoupled (+)2026.06 | 15.6 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQBackbone=LLaMA3-3B, Weight bit-width (W)=1, Activation bit-width (A)=8, Quantization Protocol=QAT2026.05 | 15.7 | — | — | — | — | — | — | — | — | — | — | |
| MoE > <formerModel Family=MoE2026.06 | 15.98 | — | — | — | — | — | — | — | — | — | — | |
| FP16 BaseBackbone=Qwen3-1.7B, Weight bit-width (W)=16, Activation bit-width (A)=16, Quantization Protocol=None (Full Precision)2026.05 | 16.2 | — | — | — | — | — | — | — | — | — | — | |
| Qronos ⊙ PiSOBackbone=Qwen-2.5, Parameters=7B, Precision=2-bit weight-only, Strategy=Interleaved (⊙)2026.06 | 16.3 | — | — | — | — | — | — | — | — | — | — | |
| 2B > <formerModel Family=Dense2026.06 | 16.32 | — | — | — | — | — | — | — | — | — | — | |
| MoE baselineModel Family=MoE2026.06 | 16.36 | — | — | — | — | — | — | — | — | — | — | |
| Qronos ⊙ PiSOBackbone=Llama-3, Parameters=8B, Precision=2-bit weight-only, Strategy=Interleaved (⊙)2026.06 | 16.6 | — | — | — | — | — | — | — | — | — | — | |
| 2B constant-widthModel Family=Dense2026.06 | 16.96 | — | — | — | — | — | — | — | — | — | — | |
| GPTQ ⊙ PiSOBackbone=Qwen-2.5, Parameters=7B, Precision=2-bit weight-only, Strategy=Interleaved (⊙)2026.06 | 17 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5BModification=SR, Replaced Layers=(15, 17, 10, 13, 14, 16, 12)2026.02 | 17.17 | — | — | — | 183.58 | 183.79 | 5,578.03 | — | — | — | — | |
| Qronos ⊙ PiSOBackbone=Qwen-2.5, Parameters=3B, Precision=2-bit weight-only, Strategy=Interleaved (⊙)2026.06 | 17.2 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQ + JacQuantBackbone=LLaMA3-1B, Weight precision=W1, Activation precision=A162026.05 | 17.6 | — | — | — | — | — | — | — | — | — | — | |
| ANTKVBackbone=LLAMA-3-8B, BIT=0.3752026.07 | 17.7 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQBackbone=LLaMA3-1B, Weight precision=W1, Activation precision=A162026.05 | 17.9 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQ + Rotation TrickBackbone=LLaMA3-1B, Weight precision=W1, Activation precision=A162026.05 | 18.1 | — | — | — | — | — | — | — | — | — | — | |
| Qronos ⊙ PiSOBackbone=Llama-3, Parameters=3B, Precision=2-bit weight-only, Strategy=Interleaved (⊙)2026.06 | 18.9 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-1.5BModification=SR, Replaced Layers=(15, 17, 10, 13, 14, 16, 12, 9)2026.02 | 19.26 | — | — | — | 179.12 | 179.48 | 5,716.79 | — | — | — | — | |
| GPT-2-LZero-shot=true, Model size=Large2026.05 | 19.93 | — | — | — | — | — | — | — | — | — | — | |
| Glauber-UL2-LZero-shot=true, Model size=Large, N value=3, Note=upper bound2026.05 | 20.35 | — | — | — | — | — | — | — | — | — | — | |
| CR-NetBackbone=LLaMA-2 1B, Pre-training Tokens=15.6B, Fine-tuning Epochs=82025.09 | 20.66 | — | — | — | — | — | — | — | 41.14 | — | — | |
| Glauber-UL2-MZero-shot=true, Model size=Medium, N value=3, Note=upper bound2026.05 | 20.98 | — | — | — | — | — | — | — | — | — | — | |
| ParetoQ + JacQuantBackbone=Qwen3-1.7B, Weight bit-width (W)=2, Activation bit-width (A)=8, Quantization Protocol=QAT2026.05 | 21.5 | — | — | — | — | — | — | — | — | — | — | |
| Glauber-UL2-LZero-shot=true, Model size=Large, N value=1, Note=upper bound2026.05 | 21.54 | — | — | — | — | — | — | — | — | — | — | |
| BeaconBackbone=Qwen-2.5, Parameters=7B, Precision=2-bit weight-only2026.06 | 21.9 | — | — | — | — | — | — | — | — | — | — |