Evaluation Benchmarks Zero-shot
71.76Average AccuracyDense
Evaluation Results
| Method | Links | |
|---|---|---|
| DenseModel=LLaMA2-13B, PR=0%, Tokens/s=1066, TI=1.00×, Latency=396.9, Speedup=1.00×2025.05 | 71.76 | |
| ParetoQTime (hours)=1777, GPU Mem. (GB)=82.4, Base Model=LLaMA-3.1-8B, Quantization=approx 2-bit2026.05 | 70.9 | |
| WINDQuantTime (hours)=49, GPU Mem. (GB)=17.1, Base Model=LLaMA-3.1-8B, Quantization=approx 2-bit2026.05 | 70.2 | |
| BF16 (dense)Model=Qwen3-8B2026.05 | 69.08 | |
| DenseModel=LLaMA-3-8B, Pruning depth (Lp)=7-layer2026.05 | 67.51 | |
| ADMM-QQuantization Scheme=SmoothQuant, Weight Bit-width=4, Activation Bit-width=8, Weight Quantization Granularity=per-channel2026.05 | 66.1 | |
| GPTQQuantization Scheme=SmoothQuant, Weight Bit-width=4, Activation Bit-width=8, Weight Quantization Granularity=per-channel2026.05 | 65.74 | |
| TRSPModel=LLaMA2-13B, PR=25%, Tokens/s=1386, TI=1.30×, Latency=298.4, Speedup=1.33×2025.05 | 65.11 | |
| DenseModel=OPT-13B, PR=0%, Tokens/s=1029, TI=1.00×, Latency=386.5, Speedup=1.00×2025.05 | 61.79 | |
| TRSPModel=OPT-13B, PR=25%, Tokens/s=1348, TI=1.31×, Latency=286.3, Speedup=1.35×2025.05 | 60.84 | |
| LLM-Streamline + Ghost LayerModel=LLaMA-3-8B, Pruning depth (Lp)=7-layer2026.05 | 60.1 | |
| FloatModel=SmolLM2 1.7B, Average Bit-Width=BF162026.06 | 59.52 | |
| ShortGPT + Ghost LayerModel=LLaMA-3-8B, Pruning depth (Lp)=7-layer2026.05 | 59.33 | |
| MXFP8Model=SmolLM2 1.7B, Average Bit-Width=8.02026.06 | 58.69 | |
| ADMM-QQuantization Scheme=SpinQuant, Weight Bit-width=4, Activation Bit-width=4, KV Cache Bit-width=4, Weight Quantization Granularity=per-channel2026.05 | 57.96 | |
| dMXModel=SmolLM2 1.7B, Average Bit-Width=7.292026.06 | 57.9 | |
| GPTQQuantization Scheme=SpinQuant, Weight Bit-width=4, Activation Bit-width=4, KV Cache Bit-width=4, Weight Quantization Granularity=per-channel2026.05 | 57.67 | |
| TRSPModel=LLaMA2-13B, PR=50%, Tokens/s=1823, TI=1.71×, Latency=216.9, Speedup=1.83×2025.05 | 57.57 | |
| dMXModel=SmolLM2 1.7B, Average Bit-Width=6.062026.06 | 57.28 | |
| NeUQITokens=∼ 0.5M×1, Backbone=LLaMA 2 7B, Quantization bits=2-bit, Group size=128, Method Variant=lightweight distillation2025.05 | 56.77 | |
| dMXModel=SmolLM2 1.7B, Average Bit-Width=5.092026.06 | 55.36 | |
| GPTQQuantization Scheme=SmoothQuant, Weight Bit-width=3, Activation Bit-width=8, Weight Quantization Granularity=per-channel2026.05 | 54.91 | |
| ADMM-QQuantization Scheme=SpinQuant, Weight Bit-width=3, Activation Bit-width=4, KV Cache Bit-width=4, Weight Quantization Granularity=per-channel2026.05 | 54.7 | |
| ADMM-QQuantization Scheme=SmoothQuant, Weight Bit-width=3, Activation Bit-width=8, Weight Quantization Granularity=per-channel2026.05 | 54.64 | |
| dMXModel=SmolLM2 1.7B, Average Bit-Width=4.532026.06 | 54.41 | |
| FloatModel=Qwen3 1.7B, Average Bit-Width=BF162026.06 | 54.12 | |
| dMXModel=Qwen3 1.7B, Average Bit-Width=7.652026.06 | 54.09 | |
| MXFP8Model=Qwen3 1.7B, Average Bit-Width=8.02026.06 | 54.08 | |
| ShortGPT + Ghost LayerModel=LLaMA-3.1-8B, Pruning depth (Lp)=11-layer2026.05 | 54.02 | |
| GPTQQuantization Scheme=SpinQuant, Weight Bit-width=3, Activation Bit-width=4, KV Cache Bit-width=4, Weight Quantization Granularity=per-channel2026.05 | 53.97 | |
| LLM-Streamline + Ghost LayerModel=LLaMA-3.1-8B, Pruning depth (Lp)=11-layer2026.05 | 53.92 | |
| dMXModel=Qwen3 1.7B, Average Bit-Width=5.902026.06 | 53.74 | |
| Shortened LLaMA + Ghost LayerModel=LLaMA-3-8B, Pruning depth (Lp)=7-layer2026.05 | 53.44 | |
| MXFP4Model=SmolLM2 1.7B, Average Bit-Width=4.02026.06 | 53.42 | |
| dMXModel=Qwen3 1.7B, Average Bit-Width=4.532026.06 | 52.2 | |
| PV-tuningTokens=∼ 1B×1, Backbone=LLaMA 2 7B, Quantization bits=2-bit, Group size=1282025.05 | 52.17 | |
| dMXModel=Qwen3 1.7B, Average Bit-Width=5.022026.06 | 51.62 | |
| FloatModel=Llama 3.2 1B, Average Bit-Width=BF162026.06 | 51.53 | |
| dMXModel=Llama 3.2 1B, Average Bit-Width=8.02026.06 | 51.46 | |
| MXFP8Model=Llama 3.2 1B, Average Bit-Width=8.02026.06 | 51.39 | |
| MXFP4Model=Qwen3 1.7B, Average Bit-Width=4.02026.06 | 51.15 | |
| TRSPModel=OPT-13B, PR=50%, Tokens/s=1801, TI=1.75×, Latency=208.9, Speedup=1.85×2025.05 | 50.83 | |
| dMXModel=Llama 3.2 1B, Average Bit-Width=6.042026.06 | 49.61 | |
| dMXModel=Llama 3.2 1B, Average Bit-Width=5.112026.06 | 49.27 | |
| dMXModel=Llama 3.2 1B, Average Bit-Width=4.572026.06 | 48.11 | |
| GPTQTokens=∼ 0.5M×1, Backbone=LLaMA 2 7B, Quantization bits=2-bit, Group size=128, Method Variant=lightweight distillation2025.05 | 48.05 | |
| MXFP4Model=Llama 3.2 1B, Average Bit-Width=4.02026.06 | 47.82 | |
| OmniQuantTokens=∼0.25M×20, Backbone=LLaMA 2 7B, Quantization bits=2-bit, Group size=1282025.05 | 47.59 | |
| ADMM-QQuantization Scheme=SpinQuant, Weight Bit-width=2, Activation Bit-width=4, KV Cache Bit-width=4, Weight Quantization Granularity=per-channel2026.05 | 46.29 | |
| ADMM-QQuantization Scheme=SmoothQuant, Weight Bit-width=2, Activation Bit-width=8, Weight Quantization Granularity=per-channel2026.05 | 46.13 | |
| EfficientQATTime (hours)=14, GPU Mem. (GB)=8.1, Base Model=LLaMA-3.1-8B, Quantization=approx 2-bit2026.05 | 44.8 | |
| GPTQQuantization Scheme=SpinQuant, Weight Bit-width=2, Activation Bit-width=4, KV Cache Bit-width=4, Weight Quantization Granularity=per-channel2026.05 | 43.28 | |
| Shortened LLaMA + Ghost LayerModel=LLaMA-3.1-8B, Pruning depth (Lp)=11-layer2026.05 | 42.31 | |
| LLM-QATTime (hours)=0.5, GPU Mem. (GB)=126.9, Base Model=LLaMA-3.1-8B, Quantization=approx 2-bit2026.05 | 42 | |
| GPTQQuantization Scheme=SmoothQuant, Weight Bit-width=2, Activation Bit-width=8, Weight Quantization Granularity=per-channel2026.05 | 41.38 |