List your company
ResearchFormerly WizWand

Language Modeling on WikiText-2 (Perplexity)

3.84PerplexityVanilla
3.55525.47767.49.3224Jun 5, 2025Aug 10, 2025Oct 15, 2025Dec 21, 2025Feb 25, 2026May 2, 2026Jul 8, 2026
Updated 3mo ago

Evaluation results

MethodLinks
VanillaBackbone=Mixtral-8x7B-v0.1, Sparsity=0%2025.113.84
PuzzleMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.114.1
PuzzleMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.114.36
D2Backbone=Mixtral-8x7B-v0.1, Sparsity=20%2025.114.65
FP16Bits (W-A-KV)=16-16-16, Model=Q2.5-32B2025.064.67
NAEEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.115.01
FlatQuantBits (W-A-KV)=4-8-8, Model=Q2.5-32B2025.065.05
FPTQuantBits (W-A-KV)=4-8-8, Model=Q2.5-32B2025.065.12
Sub-MoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.115.16
D2Backbone=Mixtral-8x7B-v0.1, Sparsity=40%2025.115.28
QuaRot-optBits (W-A-KV)=4-8-8, Model=Q2.5-32B2025.065.28
SpinQuantBits (W-A-KV)=4-8-8, Model=Q2.5-32B2025.065.28
OSTQuantBits (W-A-KV)=4-8-8, Model=Q2.5-32B2025.065.28
HC-SMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=25%2025.115.31
DenseSparsity=0%, Hardware=A100, Batch Size=1, Sequence Length=5122026.075.33
FP16Bits (W-A-KV)=16-16-16, Model=L2-7B2025.065.47
BaselineRatio=0.0, Backbone=LLaMA-7B, Fine-tuning=false2026.075.68
RTN-optBits (W-A-KV)=4-8-8, Model=Q2.5-32B2025.065.72
FP16Bits (W-A-KV)=16-16-16, Model=L3-8B2025.065.75
FPTQuantBits (W-A-KV)=4-8-8, Model=L2-7B2025.065.85
WandaBackbone=Mixtral-8x7B-v0.1, Sparsity=2:42025.115.89
SpinQuantBits (W-A-KV)=4-8-8, Model=L2-7B2025.065.97
FlatQuantBits (W-A-KV)=4-8-8, Model=L3-8B2025.066.2
QuaRot-optBits (W-A-KV)=4-8-8, Model=L2-7B2025.066.22
FPTQuantBits (W-A-KV)=4-4-4, Model=Q2.5-32B2025.066.24
FPTQuantBits (W-A-KV)=4-8-8, Model=L3-8B2025.066.27
FP16Bits (W-A-KV)=16-16-16, Model=M-8B it2025.066.45
FlatQuantBits (W-A-KV)=4-8-8, Model=L2-7B2025.066.46
NAEEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.116.49
OSTQuantBits (W-A-KV)=4-8-8, Model=L2-7B2025.066.49
VanillaBackbone=Deepseek-MoE-16b, Sparsity=0%2025.116.51
SpinQuantBits (W-A-KV)=4-8-8, Model=L3-8B2025.066.54
OSTQuantBits (W-A-KV)=4-8-8, Model=L3-8B2025.066.56
FlatQuantBits (W-A-KV)=4-4-4, Model=Q2.5-32B2025.066.57
PuzzleMoEBackbone=Deepseek-MoE-16b, Sparsity=25%2025.116.68
FlatQuantBits (W-A-KV)=4-8-8, Model=M-8B it2025.066.69
FPTQuantBits (W-A-KV)=4-8-8, Model=M-8B it2025.066.72
QuaRot-optBits (W-A-KV)=4-8-8, Model=M-8B it2025.066.76
OSTQuantBits (W-A-KV)=4-8-8, Model=M-8B it2025.066.82
RTNBits (W-A-KV)=4-8-8, Model=Q2.5-32B2025.066.83
D2Backbone=Deepseek-MoE-16b, Sparsity=20%2025.116.84
SpinQuantBits (W-A-KV)=4-8-8, Model=M-8B it2025.066.86
PuzzleMoEBackbone=Deepseek-MoE-16b, Sparsity=50%2025.116.88
Sub-MoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.116.97
QuaRot-optBits (W-A-KV)=4-8-8, Model=L3-8B2025.067.04
RTN-optBits (W-A-KV)=4-8-8, Model=L2-7B2025.067.11
VanillaBackbone=Qwen1.5-MoE-A2.7B, Sparsity=0%2025.117.22
RTN-optBits (W-A-KV)=4-8-8, Model=L3-8B2025.067.32
PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=25%2025.117.37
LACE-SVDRatio=0.2, Backbone=LLaMA-7B, Fine-tuning=false2026.077.39
FlatQuantModel=Q2.5-7B it, Quantization Setting=(i) Linear+KV, Quantization Precision (W-A-KV)=W4A8KV82025.067.47
FlatQuantModel=Q2.5-7B it, Quantization Setting=(ii) +BMM, Quantization Precision (W-A-KV)=W4A8KV82025.067.51
QuaRot-optBits (W-A-KV)=4-4-4, Model=Q2.5-32B2025.067.51
PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Sparsity=50%2025.117.55
SpinQuantBits (W-A-KV)=4-4-4, Model=Q2.5-32B2025.067.59
LACE-SVDMemory Budget=10 GB, Backbone=LLaMA-7B2026.077.6
FPTQuantModel=Q2.5-7B it, Quantization Setting=(i) Linear+KV, Quantization Precision (W-A-KV)=W4A8KV82025.067.61
HC-SMoEBackbone=Mixtral-8x7B-v0.1, Sparsity=50%2025.117.65
SpinquantModel=Q2.5-7B it, Quantization Setting=(i) Linear+KV, Quantization Precision (W-A-KV)=W4A8KV82025.067.66
FPTQuantModel=Q2.5-7B it, Quantization Setting=(ii) +BMM, Quantization Precision (W-A-KV)=W4A8KV82025.067.74
SpinquantModel=Q2.5-7B it, Quantization Setting=(ii) +BMM, Quantization Precision (W-A-KV)=W4A8KV82025.067.87
D2Backbone=Deepseek-MoE-16b, Sparsity=40%2025.117.93
SVD-LLMRatio=0.2, Backbone=LLaMA-7B, Fine-tuning=false2026.077.94
QuaRot-optBits (W-A-KV)=4-4-4, Model=M-8B it2025.068.34
OSTQuantBits (W-A-KV)=4-4-4, Model=Q2.5-32B2025.068.39
FPTQuantModel=Q2.5-7B it, Quantization Setting=(iii) All except residual, Quantization Precision (W-A-KV)=W4A8KV82025.068.44
FlatQuantBits (W-A-KV)=4-4-4, Model=M-8B it2025.068.44
WandaBackbone=Deepseek-MoE-16b, Sparsity=2:42025.118.46
Sub-MoEBackbone=Deepseek-MoE-16b, Sparsity=25%2025.118.48
FPTQuantBits (W-A-KV)=4-4-4, Model=M-8B it2025.068.49
Dobi-SVDRatio=0.2, Backbone=LLaMA-7B, Fine-tuning=false2026.078.54
FP16BIT=162026.078.55
SpinQuantBits (W-A-KV)=4-4-4, Model=M-8B it2025.068.69
VanillaBackbone=Qwen3-MoE-30B-A3B, Sparsity=0%2025.118.7
SliceGPTMemory Budget=10 GB, Backbone=LLaMA-7B2026.078.78
WandaBackbone=Qwen1.5-MoE-A2.7B, Sparsity=2:42025.118.81
LACE-SVDMemory Budget=9 GB, Backbone=LLaMA-7B2026.078.85
GSRQBIT=22026.078.88
VQLLMBIT=22026.079.05
PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=25%2025.119.08
SpinquantModel=Q2.5-7B it, Quantization Setting=(iii) All except residual, Quantization Precision (W-A-KV)=W4A8KV82025.069.23
FlatQuantModel=Q2.5-7B it, Quantization Setting=(iii) All except residual, Quantization Precision (W-A-KV)=W4A8KV82025.069.24
BlockPrunerMemory Budget=10 GB, Backbone=LLaMA-7B2026.079.4
PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Sparsity=50%2025.119.5
FlatQuantModel=L3-8B, Quantization Setting=(i) Linear+KV, Quantization Precision (W-A-KV)=W4A4KV42025.069.55
FlatQuantBits (W-A-KV)=4-4-4, Model=L3-8B2025.069.55
FP16 BaselineBackbone=LLaMA3-1B, Weight bit-width=16, Activation bit-width=16, Evaluation protocol=Zero-shot2026.059.6
OSTQuantBits (W-A-KV)=4-4-4, Model=L3-8B2025.069.66
FPTQuantModel=L3-8B, Quantization Setting=(i) Linear+KV, Quantization Precision (W-A-KV)=W4A4KV42025.069.74
FPTQuantBits (W-A-KV)=4-4-4, Model=L3-8B2025.069.74
LLM-PrunerMemory Budget=10 GB, Backbone=LLaMA-7B2026.079.88
OSTQuantBits (W-A-KV)=4-4-4, Model=M-8B it2025.0610
RTN-optBits (W-A-KV)=4-8-8, Model=M-8B it2025.0610.13
GSRQBIT=12026.0710.22
FP16Bits (W-A-KV)=16-16-16, Model=L3.2-3B it2025.0610.48
FPTQuantBits (W-A-KV)=4-8-8, Model=L3.2-3B it2025.0610.65
FlatQuantBits (W-A-KV)=4-8-8, Model=L3.2-3B it2025.0610.67
QuaRot-optBits (W-A-KV)=4-8-8, Model=L3.2-3B it2025.0610.89
D2Backbone=Qwen1.5-MoE-A2.7B, Sparsity=20%2025.1110.91
PALSSparsity=50%, Hardware=A100, Batch Size=1, Sequence Length=5122026.0710.96