List your company
ResearchFormerly WizWand

Language Modeling on C4 (Perplexity)

1PerplexityWanda
0.73682.51344.296.0666Oct 2, 2025Nov 9, 2025Dec 17, 2025Jan 24, 2026Mar 3, 2026Apr 10, 2026May 18, 2026
Updated 5mo ago

Evaluation results

MethodLinks
WandaWall-clock (h)=0.159, #GPUs=1, Model=LLaMA-2-7B, Sparsity=90%2025.101
W16A16 BaselineBits=W16A16, Model=Llama2-70B2026.035.54
SliderQuantBits=W4A16, Model=Llama2-70B2026.035.6
CBQBits=W4A16, Model=Llama2-70B2026.035.64
OmniQuantBits=W4A16, Model=Llama2-70B2026.035.65
GPTQBits=W4A16, Model=Llama2-70B2026.035.67
AWQBits=W4A16, Model=Llama2-70B2026.035.7
FP16#Bits W-A-KV=4-4-4, Model=LLaMA2-70B2026.055.72
RTNBits=W4A16, Model=Llama2-70B2026.035.79
BDQ#Bits W-A-KV=4-4-4, Model=LLaMA2-70B2026.055.88
FlatQuant#Bits W-A-KV=4-4-4, Model=LLaMA2-70B2026.055.92
SpinQuant#Bits W-A-KV=4-4-4, Model=LLaMA2-70B2026.056.07
QuaRot#Bits W-A-KV=4-4-4, Model=LLaMA2-70B2026.056.12
FPBackbone=Llama2-13b, Weight Bits=Full precision, Activation Bits=Full precision2026.056.41
FPModel=Llama2-13b, W/A=FP2026.056.41
W16A16 BaselineBits=W16A16, Model=Llama2-13B2026.036.46
BaselinePrecision=FP16, Model=LLaMA2-13B2026.056.46
BaselinePrec.=FP16, Model Scale=LLaMA2-13B2026.056.46
FP16Model=Llama-2-13B2026.036.47
BF16Backbone=Llama-2-13B, Quantization=None2026.056.47
SliderQuantBits=W4A16, Model=Llama2-13B2026.036.54
OmniQuantBits=W4A16, Model=Llama2-13B2026.036.65
CBQBits=W4A16, Model=Llama2-13B2026.036.67
GPTQBits=W4A16, Model=Llama2-13B2026.036.7
DenseRatio=1.0, zero-shot evaluation=true2026.046.73
FP16Model=L2-13B, Calibration samples=1282026.046.73
FP16Model=L2-13B, Bits=16-bit2026.046.73
FP16Precision=FP16, Model Architecture=L2-13B2026.046.73
FP16Model=L2-13B2026.046.73
LLaMA2-13BBackbone=LLaMA2-13B, Bit=-, T × D=-, R=-, FLOPs(T)=13.42, Power(J)=61.742026.046.73
FP16#Bits W-A-KV=4-4-4, Model=LLaMA2-13B2026.056.73
AWQBits=W4A16, Model=Llama2-13B2026.036.74
OSAQ+WKVQuantPrecision=W4A16KV4, Model=LLaMA2-13B2026.056.81
RTNBits=W4A16, Model=Llama2-13B2026.036.83
OSAQ+QuaRotPrec.=W4A4, Model Scale=LLaMA2-13B2026.056.85
OSAQ+SpinQuantPrec.=W4A4, Model Scale=LLaMA2-13B2026.056.86
WKVQuantPrecision=W4A16KV4, Model=LLaMA2-13B2026.056.89
FPBackbone=Llama2-7b, Weight Bits=Full precision, Activation Bits=Full precision2026.056.9
FPModel=Llama2-7b, W/A=FP2026.056.9
SpinQuantPrec.=W4A4, Model Scale=LLaMA2-13B2026.056.93
OSAQ+DuQuantPrec.=W4A4, Model Scale=LLaMA2-13B2026.056.96
W16A16 BaselineBits=W16A16, Model=Llama2-7B2026.036.97
FP16Model Architecture=Llama-2-7B2026.046.97
LLAMA-2-7BModel=LLAMA-2-7B, Bits=FP162026.046.97
BaselinePrecision=FP16, Model=LLaMA2-7B2026.056.97
BaselinePrec.=FP16, Model Scale=LLaMA2-7B2026.056.97
QuaRotPrec.=W4A4, Model Scale=LLaMA2-13B2026.056.97
BF16Backbone=LLaMA-2-7B, Quantization=None2026.056.97
DuQuantPrec.=W4A4, Model Scale=LLaMA2-13B2026.057.05
FP16Bits (W-A-KV)=4-4-4, Model=DeepSeek-R1-Distill LLaMA-70B, Zero-shot=true2026.057.06
FP16Bits (W-A-KV)=2-4-16, Model=DeepSeek-R1-Distill LLaMA-70B, Zero-shot=true2026.057.06
BDQ#Bits W-A-KV=4-4-4, Model=LLaMA2-13B2026.057.07
LLAMA-1-7BModel=LLAMA-1-7B, Bits=FP162026.047.08
FlatQuant#Bits W-A-KV=4-4-4, Model=LLaMA2-13B2026.057.11
SDLLMBackbone=LLaMA2-13B, Bit=4-1.58, T × D=1 × 16, R=0.215, FLOPs(T)=2.89, Power(J)=2.602026.047.15
FP16Model=L3-70B, Bits=16-bit2026.047.17
FP16Precision=FP16, Model Architecture=L3-70B2026.047.17
FP16Model=L3-70B2026.047.17
GPTAQ-MARRModel=Llama2-13b, W/A=W4A42026.057.17
FP16#Bits W-A-KV=4-4-4, Model=LLaMA3-70B2026.057.17
FP16#Bits W-A-KV=3-3-3, Model=LLaMA3-70B2026.057.17
FP16#Bits W-A-KV=2-4-16, Model=LLaMA3-70B2026.057.17
FP16Bits (W-A-KV)=4-4-4, Model=LLaMA3-70B, Zero-shot=true2026.057.17
FP16Bits (W-A-KV)=2-4-16, Model=LLaMA3-70B, Zero-shot=true2026.057.17
SliderQuantBits=W4A16, Model=Llama2-7B2026.037.19
GPTAQModel=Llama2-13b, W/A=W4A42026.057.22
CBQBits=W4A16, Model=Llama2-7B2026.037.23
DenseSparsity=0%, Backbone=LLaMA-2-7B2025.107.26
FP16Model=L2-7B, Calibration samples=1282026.047.26
FP16Model=L3-8B, Calibration samples=1282026.047.26
FP16Model=L2-7B, Bits=16-bit2026.047.26
FP16Precision=FP16, Model Architecture=L2-7B2026.047.26
FP16Model=L2-7B2026.047.26
LLaMA2-7BBackbone=LLaMA2-7B, Bit=-, T × D=-, R=-, FLOPs(T)=6.91, Power(J)=31.772026.047.26
ResComp-MARRModel=Llama2-13b, W/A=W4A42026.057.26
FP16#Bits W-A-KV=4-4-4, Model=LLaMA2-7B2026.057.26
OriginalParam Ratio=1.0, Backbone=LLaMA-2-7B2026.047.2745
ResCompModel=Llama2-13b, W/A=W4A42026.057.28
OmniQuantPrecision=W4A16KV4, Model=LLaMA2-13B2026.057.3
BaselineRatio=1.0, Memory Footprint (GB)=12.6GB2026.047.34
DenseCompression Ratio=1.0, Backbone=LLaMA-7B2026.047.34
GPTAQModel=L2-13B, Bits=3-bit, Group Size=g1282026.047.34
GPTAQ+OursModel=L2-13B, Bits=3-bit, Group Size=g1282026.047.34
BaselineCompression Ratio=0.02026.057.34
OmniQuantBits=W4A16, Model=Llama2-7B2026.037.35
GPTQBits=W4A16, Model=Llama2-7B2026.037.37
GPTQ+OursModel=L2-13B, Bits=3-bit, Group Size=g1282026.047.37
GPTQModel=Llama2-13b, W/A=W4A42026.057.37
GPTQModel=L2-13B, Bits=3-bit, Group Size=g1282026.047.38
BF16Models=Mixtral-8x7B, Quantization Strategy=Embedding-wise, Quantization Bit-width=BF162026.047.41
OursBits (W-A-KV)=4-4-4, Model=DeepSeek-R1-Distill LLaMA-70B, Zero-shot=true2026.057.41
OSAQ+WKVQuantPrecision=W4A16KV4, Model=LLaMA2-7B2026.057.43
GPTAQ-MARRBackbone=Llama2-13b, Weight Bits=3, Activation Bits=162026.057.45
SpinQuant#Bits W-A-KV=4-4-4, Model=LLaMA2-13B2026.057.48
WKVQuantPrecision=W4A16KV4, Model=LLaMA2-7B2026.057.49
GPTAQBackbone=Llama2-13b, Weight Bits=3, Activation Bits=162026.057.5
ResComp-MARRBackbone=Llama2-13b, Weight Bits=3, Activation Bits=162026.057.54
QuaRot#Bits W-A-KV=4-4-4, Model=LLaMA2-13B2026.057.54
AWQModel=L2-13B, Bits=3-bit, Group Size=g1282026.047.57
ResCompBackbone=Llama2-13b, Weight Bits=3, Activation Bits=162026.057.58