Language Modeling on WikiText-2, PTB, and C4
6.19PerplexityFloatingPoint
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| FloatingPointW-A-KV Bits=16-16-16, Backbone=Llama-3 70B2026.05 | 6.19 | — | — | — | |
| ConQuR (f.p. calib.)W-A-KV Bits=4-4-16, Backbone=Llama-3 70B, Calibration Protocol=Full-precision2026.05 | 8.25 | — | — | — | |
| ConQuR (quant. calib.)W-A-KV Bits=4-4-16, Backbone=Llama-3 70B, Calibration Protocol=Quantization-aware2026.05 | 8.29 | — | — | — | |
| DartQuantW-A-KV Bits=4-4-16, Backbone=Llama-3 70B2026.05 | 8.31 | — | — | — | |
| ConQuR (f.p. calib.)W-A-KV Bits=4-4-4, Backbone=Llama-3 70B, Calibration Protocol=Full-precision2026.05 | 8.4 | — | — | — | |
| DenseBackbone=LLaMA3-8B, Pruning Layers=02026.05 | 8.44 | 6.14 | 10.58 | 8.61 | |
| DartQuantW-A-KV Bits=4-4-4, Backbone=Llama-3 70B2026.05 | 8.46 | — | — | — | |
| ConQuR (quant. calib.)W-A-KV Bits=4-4-4, Backbone=Llama-3 70B, Calibration Protocol=Quantization-aware2026.05 | 8.49 | — | — | — | |
| DenseBackbone=LLaMA3.1-8B, Pruning Layers=02026.05 | 8.5 | 6.24 | 10.58 | 8.68 | |
| DenseBackbone=LLaMA-3.1-8B, Lp/Lt=7/322026.05 | 8.5 | 6.24 | 10.58 | 8.68 | |
| DenseBackbone=LLaMA-3.1-8B, Lp/Lt=9/322026.05 | 8.5 | 6.24 | 10.58 | 8.68 | |
| FloatingPointW-A-KV Bits=16-16-16, Backbone=Llama-3 8B2026.05 | 8.92 | — | — | — | |
| SpinQuantW-A-KV Bits=4-4-4, Backbone=Llama-3 70B2026.05 | 9.06 | — | — | — | |
| SpinQuantW-A-KV Bits=4-4-16, Backbone=Llama-3 70B2026.05 | 9.31 | — | — | — | |
| ConQuR (quant. calib.)W-A-KV Bits=4-4-16, Backbone=Llama-3 8B, Calibration Protocol=Quantization-aware2026.05 | 10.62 | — | — | — | |
| ConQuR (f.p. calib.)W-A-KV Bits=4-4-16, Backbone=Llama-3 8B, Calibration Protocol=Full-precision2026.05 | 10.7 | — | — | — | |
| DartQuantW-A-KV Bits=4-4-16, Backbone=Llama-3 8B2026.05 | 10.79 | — | — | — | |
| ConQuR (quant. calib.)W-A-KV Bits=4-4-4, Backbone=Llama-3 8B, Calibration Protocol=Quantization-aware2026.05 | 10.81 | — | — | — | |
| ConQuR (f.p. calib.)W-A-KV Bits=4-4-4, Backbone=Llama-3 8B, Calibration Protocol=Full-precision2026.05 | 10.88 | — | — | — | |
| FloatingPointW-A-KV Bits=16-16-16, Backbone=Llama-3.2 3B2026.05 | 10.9 | — | — | — | |
| DFRotW-A-KV Bits=4-4-16, Backbone=Llama-3 8B2026.05 | 10.96 | — | — | — | |
| DartQuantW-A-KV Bits=4-4-4, Backbone=Llama-3 8B2026.05 | 11 | — | — | — | |
| QuaRotW-A-KV Bits=4-4-16, Backbone=Llama-3 8B2026.05 | 11.06 | — | — | — | |
| SpinQuantW-A-KV Bits=4-4-16, Backbone=Llama-3 8B2026.05 | 11.13 | — | — | — | |
| DFRotW-A-KV Bits=4-4-4, Backbone=Llama-3 8B2026.05 | 11.19 | — | — | — | |
| SpinQuantW-A-KV Bits=4-4-4, Backbone=Llama-3 8B2026.05 | 11.28 | — | — | — | |
| QuaRotW-A-KV Bits=4-4-4, Backbone=Llama-3 8B2026.05 | 11.31 | — | — | — | |
| DenseBackbone=LLaMA-2-7B, Pruning Layers=02026.05 | 11.56 | 5.47 | 22.51 | 6.71 | |
| DenseBackbone=Qwen3-14B, Lp/Lt=11/402026.05 | 12.14 | 8.64 | 14.79 | 13 | |
| DenseBackbone=Qwen3-14B, Lp/Lt=13/402026.05 | 12.14 | 8.64 | 14.79 | 13 | |
| ConQuR (quant. calib.)W-A-KV Bits=4-4-16, Backbone=Llama-3.2 3B, Calibration Protocol=Quantization-aware2026.05 | 13.21 | — | — | — | |
| QuaRotW-A-KV Bits=4-4-16, Backbone=Llama-3 70B2026.05 | 13.26 | — | — | — | |
| ConQuR (f.p. calib.)W-A-KV Bits=4-4-16, Backbone=Llama-3.2 3B, Calibration Protocol=Full-precision2026.05 | 13.29 | — | — | — | |
| DartQuantW-A-KV Bits=4-4-16, Backbone=Llama-3.2 3B2026.05 | 13.37 | — | — | — | |
| QuaRotW-A-KV Bits=4-4-4, Backbone=Llama-3 70B2026.05 | 13.53 | — | — | — | |
| SpinQuantW-A-KV Bits=4-4-16, Backbone=Llama-3.2 3B2026.05 | 13.65 | — | — | — | |
| ConQuR (quant. calib.)W-A-KV Bits=4-4-4, Backbone=Llama-3.2 3B, Calibration Protocol=Quantization-aware2026.05 | 13.68 | — | — | — | |
| DenseBackbone=Qwen3-8B, Lp/Lt=7/362026.05 | 13.69 | 9.71 | 16.9 | 14.45 | |
| DenseBackbone=Qwen3-8B, Lp/Lt=9/362026.05 | 13.69 | 9.71 | 16.9 | 14.45 | |
| ConQuR (f.p. calib.)W-A-KV Bits=4-4-4, Backbone=Llama-3.2 3B, Calibration Protocol=Full-precision2026.05 | 13.74 | — | — | — | |
| QuaRotW-A-KV Bits=4-4-16, Backbone=Llama-3.2 3B2026.05 | 13.75 | — | — | — | |
| DFRotW-A-KV Bits=4-4-16, Backbone=Llama-3.2 3B2026.05 | 13.76 | — | — | — | |
| DenseBackbone=Mistral-Nemo-12B, Lp/Lt=11/402026.05 | 13.85 | 5.75 | 26.7 | 9.09 | |
| DenseBackbone=Mistral-Nemo-12B, Lp/Lt=13/402026.05 | 13.85 | 5.75 | 26.7 | 9.09 | |
| DartQuantW-A-KV Bits=4-4-4, Backbone=Llama-3.2 3B2026.05 | 13.86 | — | — | — | |
| SpinQuantW-A-KV Bits=4-4-4, Backbone=Llama-3.2 3B2026.05 | 13.98 | — | — | — | |
| QuaRotW-A-KV Bits=4-4-4, Backbone=Llama-3.2 3B2026.05 | 14.23 | — | — | — | |
| DFRotW-A-KV Bits=4-4-4, Backbone=Llama-3.2 3B2026.05 | 14.24 | — | — | — | |
| DenseBackbone=OLMo-3-7B, Lp/Lt=7/322026.05 | 14.34 | 9.92 | 17.4 | 15.7 | |
| DenseBackbone=OLMo-3-7B, Lp/Lt=9/322026.05 | 14.34 | 9.92 | 17.4 | 15.7 | |
| FloatingPointW-A-KV Bits=16-16-16, Backbone=Llama-2 7B2026.05 | 16.88 | — | — | — | |
| Ghost Layer + FTBackbone=LLaMA3-8B, Pruning Layers=72026.05 | 17 | 13 | 23.03 | 14.96 | |
| Ghost Layer + FTBackbone=LLaMA3.1-8B, Pruning Layers=72026.05 | 17.08 | 13.1 | 23.14 | 14.99 | |
| Ghost Layer + FTBackbone=LLaMA-2-7B, Pruning Layers=72026.05 | 17.2 | 10.01 | 31.27 | 10.31 | |
| DenseBackbone=D-LLaMA-8B, Pruning Layers=02026.05 | 18.29 | 13.13 | 22.28 | 19.46 | |
| ConQuR (quant. calib.)W-A-KV Bits=4-4-4, Backbone=Llama-2 7B, Calibration Protocol=Quantization-aware2026.05 | 18.64 | — | — | — | |
| DFRotW-A-KV Bits=4-4-16, Backbone=Llama-2 7B2026.05 | 18.66 | — | — | — | |
| ConQuR (quant. calib.)W-A-KV Bits=4-4-16, Backbone=Llama-2 7B, Calibration Protocol=Quantization-aware2026.05 | 18.77 | — | — | — | |
| DartQuantW-A-KV Bits=4-4-16, Backbone=Llama-2 7B2026.05 | 18.88 | — | — | — | |
| ConQuR (f.p. calib.)W-A-KV Bits=4-4-4, Backbone=Llama-2 7B, Calibration Protocol=Full-precision2026.05 | 19.15 | — | — | — | |
| DFRotW-A-KV Bits=4-4-4, Backbone=Llama-2 7B2026.05 | 19.18 | — | — | — | |
| ConQuR (f.p. calib.)W-A-KV Bits=4-4-16, Backbone=Llama-2 7B, Calibration Protocol=Full-precision2026.05 | 19.28 | — | — | — | |
| DartQuantW-A-KV Bits=4-4-4, Backbone=Llama-2 7B2026.05 | 19.57 | — | — | — | |
| SpinQuantW-A-KV Bits=4-4-16, Backbone=Llama-2 7B2026.05 | 19.97 | — | — | — | |
| Linear Patch (Rotate) + FTBackbone=LLaMA-2-7B, Pruning Layers=72026.05 | 20.09 | 10.91 | 37.28 | 12.07 | |
| Linear Patch (Diag) + FTBackbone=LLaMA-2-7B, Pruning Layers=72026.05 | 20.14 | 11.13 | 37.09 | 12.19 | |
| QuaRotW-A-KV Bits=4-4-16, Backbone=Llama-2 7B2026.05 | 20.34 | — | — | — | |
| SpinQuantW-A-KV Bits=4-4-4, Backbone=Llama-2 7B2026.05 | 20.69 | — | — | — | |
| FloatingPointW-A-KV Bits=16-16-16, Backbone=Llama-2 13B2026.05 | 20.85 | — | — | — | |
| QuaRotW-A-KV Bits=4-4-4, Backbone=Llama-2 7B2026.05 | 21.08 | — | — | — | |
| ConQuR (f.p. calib.)W-A-KV Bits=4-4-16, Backbone=Llama-2 13B, Calibration Protocol=Full-precision2026.05 | 22.78 | — | — | — | |
| Linear Patch (Rotate) + FTBackbone=LLaMA3.1-8B, Pruning Layers=72026.05 | 22.97 | 17.05 | 30.33 | 21.54 | |
| Linear Patch (Diag) + FTBackbone=LLaMA3.1-8B, Pruning Layers=72026.05 | 22.99 | 17.26 | 30.2 | 21.51 | |
| ConQuR (f.p. calib.)W-A-KV Bits=4-4-4, Backbone=Llama-2 13B, Calibration Protocol=Full-precision2026.05 | 23.02 | — | — | — | |
| Linear Patch (Diag) + FTBackbone=LLaMA3-8B, Pruning Layers=72026.05 | 23.04 | 17.29 | 30.25 | 21.59 | |
| Linear Patch (Rotate) + FTBackbone=LLaMA3-8B, Pruning Layers=72026.05 | 23.08 | 17.15 | 30.48 | 21.62 | |
| ConQuR (quant. calib.)W-A-KV Bits=4-4-4, Backbone=Llama-2 13B, Calibration Protocol=Quantization-aware2026.05 | 23.12 | — | — | — | |
| ConQuR (quant. calib.)W-A-KV Bits=4-4-16, Backbone=Llama-2 13B, Calibration Protocol=Quantization-aware2026.05 | 23.24 | — | — | — | |
| DFRotW-A-KV Bits=4-4-16, Backbone=Llama-2 13B2026.05 | 23.5 | — | — | — | |
| DFRotW-A-KV Bits=4-4-4, Backbone=Llama-2 13B2026.05 | 23.57 | — | — | — | |
| SpinQuantW-A-KV Bits=4-4-16, Backbone=Llama-2 13B2026.05 | 23.75 | — | — | — | |
| SpinQuantW-A-KV Bits=4-4-4, Backbone=Llama-2 13B2026.05 | 24.05 | — | — | — | |
| DartQuantW-A-KV Bits=4-4-16, Backbone=Llama-2 13B2026.05 | 24.34 | — | — | — | |
| DartQuantW-A-KV Bits=4-4-4, Backbone=Llama-2 13B2026.05 | 24.57 | — | — | — | |
| QuaRotW-A-KV Bits=4-4-16, Backbone=Llama-2 13B2026.05 | 24.82 | — | — | — | |
| LoRPBackbone=OLMo-3-7B, Lp/Lt=7/322026.05 | 24.82 | 18.66 | 29.94 | 25.85 | |
| QuaRotW-A-KV Bits=4-4-4, Backbone=Llama-2 13B2026.05 | 25.17 | — | — | — | |
| Ghost Layer + FTBackbone=D-LLaMA-8B, Pruning Layers=72026.05 | 27.28 | 22.92 | 33.66 | 25.26 | |
| LoRPBackbone=Qwen3-8B, Lp/Lt=7/362026.05 | 27.92 | 22.18 | 34.22 | 27.38 | |
| LaCoBackbone=OLMo-3-7B, Lp/Lt=7/322026.05 | 29.76 | 21.84 | 39.81 | 27.63 | |
| ShortGPTBackbone=OLMo-3-7B, Lp/Lt=7/322026.05 | 30.02 | 24.25 | 35.55 | 30.24 | |
| LLM-StreamlineBackbone=OLMo-3-7B, Lp/Lt=7/322026.05 | 30.02 | 24.25 | 35.55 | 30.24 | |
| Pruned LLMBackbone=LLaMA-2-7B, Pruning Layers=72026.05 | 35.33 | 18.45 | 62.18 | 25.37 | |
| LoRPBackbone=Qwen3-14B, Lp/Lt=11/402026.05 | 37.44 | 27.11 | 51.73 | 33.47 | |
| DFRotW-A-KV Bits=4-4-16, Backbone=Llama-3 70B2026.05 | 39 | — | — | — | |
| Linear Patch (Rotate) + FTBackbone=D-LLaMA-8B, Pruning Layers=72026.05 | 39.5 | 33.64 | 51.39 | 33.47 | |
| Linear Patch (Diag) + FTBackbone=D-LLaMA-8B, Pruning Layers=72026.05 | 40.53 | 35.96 | 51.01 | 34.63 | |
| LoRPBackbone=OLMo-3-7B, Lp/Lt=9/322026.05 | 40.69 | 29.27 | 57.4 | 35.39 | |
| DFRotW-A-KV Bits=4-4-4, Backbone=Llama-3 70B2026.05 | 40.75 | — | — | — | |
| LoRPBackbone=LLaMA-3.1-8B, Lp/Lt=7/322026.05 | 41.95 | 27.88 | 62.56 | 35.41 |