List your company
ResearchFormerly WizWand

Language Modeling on C4 (Loss)

2.55C4 LossOriginal Model (no compression)
2.14244.89377.64510.3963Nov 13, 2025Dec 5, 2025Dec 27, 2025Jan 18, 2026Feb 9, 2026Mar 3, 2026Mar 26, 2026
Updated 7mo ago

Evaluation results

MethodLinks
Original Model (no compression)Model=Qwen2-7B2026.012.55—
AQLMModel=Qwen2-7B, Compression level=High, Bits per parameter (BPP)=3.01, Training budget=50 million tokens2026.012.74—
VQ-then-trainModel=Qwen2-7B, Compression level=High, Bits per parameter (BPP)=3.01, Training budget=50 million tokens2026.012.81—
VQ-then-trainModel=Qwen2-7B, Compression level=Medium, Bits per parameter (BPP)=2.26, Training budget=50 million tokens2026.013.16—
VQ-then-trainModel=Qwen2-7B, Compression level=Low, Bits per parameter (BPP)=1.5, Training budget=50 million tokens2026.013.22—
Original Model (no compression)Model=Qwen2-1.5B2026.013.28—
AQLMModel=Qwen2-7B, Compression level=Medium, Bits per parameter (BPP)=2.26, Training budget=50 million tokens2026.013.28—
VQ-then-trainModel=Qwen2-1.5B, Compression level=High, Bits per parameter (BPP)=2.03, Training budget=1 billion tokens2026.013.33—
AQLMModel=Qwen2-7B, Compression level=Low, Bits per parameter (BPP)=1.5, Training budget=50 million tokens2026.013.37—
AQLMModel=Qwen2-1.5B, Compression level=High, Bits per parameter (BPP)=2.03, Training budget=1 billion tokens2026.013.65—
VQ-then-trainModel=Qwen2-1.5B, Compression level=Medium, Bits per parameter (BPP)=1.15, Training budget=1 billion tokens2026.013.69—
VQ-then-trainModel=Qwen2-1.5B, Compression level=Low, Bits per parameter (BPP)=0.58, Training budget=1 billion tokens2026.014.27—
FP16Model=Llama-13B, #Bits=W16A162026.035.62—
Random WeightsModel=Qwen2-7B2026.016—
Random WeightsModel=Qwen2-1.5B2026.016.07—
AQLMModel=Qwen2-1.5B, Compression level=Medium, Bits per parameter (BPP)=1.15, Training budget=1 billion tokens2026.016.39—
FP16Model=L2-13B, Bits=16, Group size=1282026.026.47—
GPTAQModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=1019.22026.026.57—
SNRQ-CModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=726.12026.026.57—
SNRQ-SModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=709.42026.026.57—
LDLQModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=623.92026.026.58—
GPTQModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=727.72026.026.59—
SpQRModel=Llama-13B, #Bits=W3.45A162026.036.72—
SliderQuantModel=Llama-13B, #Bits=W3A162026.036.78—
FP16Model=L2-7B, Bits=16, Group size=1282026.026.97—
FP16Format=FP16, Quantization=None2025.116.97—
SNRQ-S+RHTModel=L2-13B, Quantization Time (s)=970.22026.027.03—
SNRQ-C+RHTModel=L2-13B, Quantization Time (s)=1106.32026.027.05—
SNRQ-SModel=L2-13B, Quantization Time (s)=786.42026.027.07—
GPTAQ+RHTModel=L2-13B, Quantization Time (s)=1259.42026.027.07—
SNRQ-SModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=724.22026.027.07—
FP16Model=Llama-7B, #Bits=W16A162026.037.08—
GPTAQModel=L2-13B, Quantization Time (s)=1012.02026.027.1—
SNRQ-CModel=L2-13B, Quantization Time (s)=965.02026.027.1—
GPTAQModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=10122026.027.1—
SNRQ-CModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=749.62026.027.1—
SNRQ-CModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=450.92026.027.12—
SNRQ-SModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=405.62026.027.12—
LDLQModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=719.22026.027.12—
SliderQuantModel=Llama-7B, #Bits=W3A162026.037.13—
GPTAQModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=610.52026.027.14—
QuIPModel=L2-13B, Quantization Time (s)=1096.22026.027.15—
GPTQModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=481.62026.027.16—
LDLQModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=404.82026.027.16—
GPTQModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=820.52026.027.16—
SpQRModel=Llama-7B, #Bits=W3.45A162026.037.28—
ParoQuantFormat=MXFP4, Quantization=W4A4, Weight Quantization=GPTQ2025.117.44—
MR-GPTQFormat=MXFP4, Quantization=W4A4, Weight Quantization=GPTQ2025.117.71—
SNRQ-S+RHTModel=L2-7B, Quantization Time (s)=644.52026.027.77—
SNRQ-C+RHTModel=L2-7B, Quantization Time (s)=697.32026.027.82—
GPTAQ+RHTModel=L2-7B, Quantization Time (s)=795.72026.027.86—
SNRQ-SModel=L2-7B, Quantization Time (s)=449.02026.027.88—
SNRQ-SModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=417.32026.027.88—
SNRQ-CModel=L2-7B, Quantization Time (s)=517.12026.027.89—
SNRQ-CModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=407.42026.027.89—
QuIPModel=L2-7B, Quantization Time (s)=704.82026.028.02—
GPTAQModel=L2-7B, Quantization Time (s)=590.42026.028.04—
GPTAQModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=590.42026.028.04—
LDLQModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=427.82026.028.12—
GPTQModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=467.92026.028.26—
AQLMModel=Qwen2-1.5B, Compression level=Low, Bits per parameter (BPP)=0.58, Training budget=1 billion tokens2026.018.49—
FP16Model=L3-8B, Bits=16, Group size=1282026.028.93—
Original (Dense)Model=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.039.05—
Original (Dense)Model=DeepSeekMoE-16B-Base, Sparsity Ratio=40%2026.039.05—
Original (Dense)Model=DeepSeekMoE-16B-Base, Sparsity Ratio=60%2026.039.05—
OursModel=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.039.38—
SNRQ-CModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=495.22026.029.44—
SNRQ-SModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=498.72026.029.44—
GPTAQModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=703.22026.029.49—
LDLQModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=425.12026.029.52—
GPTQModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=4892026.029.53—
Camera-PModel=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.039.84—
HEAPrModel=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.039.85—
NAEEModel=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.0310.07—
OursModel=DeepSeekMoE-16B-Base, Sparsity Ratio=40%2026.0310.75—
SNRQ-S+RHTModel=L3-8B, Quantization Time (s)=688.72026.0211.37—
SNRQ-C+RHTModel=L3-8B, Quantization Time (s)=741.82026.0211.58—
OursModel=Qwen3-30B-A3B, Sparsity Ratio=20%2026.0311.58—
GPTAQ+RHTModel=L3-8B, Quantization Time (s)=853.12026.0211.61—
Camera-PModel=DeepSeekMoE-16B-Base, Sparsity Ratio=40%2026.0311.68—
SNRQ-SModel=L3-8B, Quantization Time (s)=542.82026.0211.73—
SNRQ-SModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=484.72026.0211.73—
QuIPModel=L3-8B, Quantization Time (s)=763.82026.0211.8—
SNRQ-CModel=L3-8B, Quantization Time (s)=594.42026.0211.87—
SNRQ-CModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=492.32026.0211.87—
Baseline#Bits=W16A162026.0312.08—
OursModel=Qwen3-30B-A3B, Sparsity Ratio=40%2026.0312.1—
Original (Dense)Model=Qwen3-30B-A3B, Sparsity Ratio=20%2026.0312.13—
Original (Dense)Model=Qwen3-30B-A3B, Sparsity Ratio=40%2026.0312.13—
Original (Dense)Model=Qwen3-30B-A3B, Sparsity Ratio=60%2026.0312.13—
GPTAQModel=L3-8B, Quantization Time (s)=715.02026.0212.15—
GPTAQModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=7152026.0212.15—
Camera-PModel=Qwen3-30B-A3B, Sparsity Ratio=20%2026.0312.25—
HEAPrModel=DeepSeekMoE-16B-Base, Sparsity Ratio=40%2026.0312.34—
NAEEModel=Qwen3-30B-A3B, Sparsity Ratio=20%2026.0312.44—
SliderQuant#Bits=W4A162026.0312.47—
OmniQuant#Bits=W4A162026.0312.54—
D2-MoEModel=DeepSeekMoE-16B-Base, Sparsity Ratio=20%2026.0312.62—
OursModel=DeepSeekMoE-16B-Base, Sparsity Ratio=60%2026.0312.65—
GPTQModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=547.52026.0212.74—