List your company
ResearchFormerly WizWand

Question Answering on PIQA (Accuracy)

86.5AccuracyMashup Learning
79.043280.979182.91584.8509May 19, 2025Jul 23, 2025Sep 27, 2025Dec 1, 2025Feb 5, 2026Apr 11, 2026Jun 16, 2026
Updated 3mo ago

Evaluation results

MethodLinks
Mashup LearningBackbone=Mistral-7B-Instruct-v0.2, Setup=Lots-of-LoRAs2026.0386.5
BF16Models=Mixtral-8x7B, Quantization Strategy=Embedding-wise, Quantization Bit-width=BF162026.0485.6
From scratchBackbone=Mistral-7B-Instruct-v0.2, Setup=Lots-of-LoRAs2026.0385.2
Full-precisionModel=Mixtral 8x22B, Avg. bits/exp.=16 (FP), Memory (GB)=281.2, Evaluation protocol=zero-shot2026.0485.12
Text-to-LoRA used as initBackbone=Mistral-7B-Instruct-v0.2, Setup=Lots-of-LoRAs2026.0385.1
Full-precisionModel=Mixtral 8x7B, Avg. bits/exp.=16 (FP), Memory (GB)=96.8, Evaluation protocol=zero-shot2026.0483.68
Gemma 2 9BModel=Gemma 2 9B2026.0383
Focus (Gemma 2 9B)Backbone=Gemma 2 9B, K=4, dg=162026.0383
FPModel=LLaDA-1.5, Quantization=Full Precision2026.0682.97
FPModel=LLaDA-Instruct, Quantization=Full Precision2026.0682.86
CodeQuantModels=Mixtral-8x7B, Quantization Strategy=Embedding-wise, Quantization Bit-width=A4W42026.0482.7
FAIR-CalibModel=LLaDA-1.5, Quantization=W4A42026.0682.7
Mistral 7BModel=Mistral 7B2026.0382.6
Focus (Mistral 7B)Backbone=Mistral 7B, K=4, dg=162026.0382.6
ICRBackbone=Qwen2.5-7B2025.0982.6
LLaMA-2 70BModel=LLaMA-2 70B2026.0382.4
Focus (LLaMA-2 70B)Backbone=LLaMA-2 70B, K=4, dg=162026.0382.4
UniformModel=Mixtral 8x7B, Avg. bits/exp.=3, Memory (GB)=19.3, Evaluation protocol=zero-shot2026.0482.32
FAIR-CalibModel=LLaDA-Instruct, Quantization=W4A42026.0682.1
PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.75, Memory (GB)=17.7, Evaluation protocol=zero-shot2026.0482.05
STLBase Model=LLaMA-3-8B2026.0282
STL-augBase Model=LLaMA-3-8B2026.0282
SCANSBase Model=LLaMA-3-8B2026.0282
Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.75, Memory (GB)=17.7, Evaluation protocol=zero-shot2026.0481.83
FlatQuantModel=LLaDA-Instruct, Quantization=W4A42026.0681.83
BaselineBit width (b)=16.002025.0581.8
PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.625, Memory (GB)=16.9, Evaluation protocol=zero-shot2026.0481.56
Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.625, Memory (GB)=16.9, Evaluation protocol=zero-shot2026.0481.45
UniformModel=Mixtral 8x22B, Avg. bits/exp.=3, Memory (GB)=57.5, Evaluation protocol=zero-shot2026.0481.45
MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.0681.44
FlatQuantModel=LLaDA-1.5, Quantization=W4A42026.0681.28
QuaRotModel=LLaDA-Instruct, Quantization=W4A42026.0681.23
OursQBackbone=Qwen3-MoE-30B-A3B, Type=P25% Q4b, Storage=11.64GB, Evaluation Protocol=zero-shot2026.0681.18
SurgicalBase Model=LLaMA-3-8B2026.0281
OLMo-2 7BModel=OLMo-2 7B2026.0381
Focus (OLMo-2 7B)Backbone=OLMo-2 7B, K=4, dg=162026.0381
SARQC-GBS(Saliency)Model=LLaMA-30B, Precision=W4A16, Zero-shot=true2026.0580.96
FP16Model=LLaMA3.1-8B-Instruct, Precision=MXFP42026.0480.9
LLaMA-3 8BPruning Ratio=0%, Base Model=LLaMA-3 8B, Evaluation Protocol=Zero-shot2026.0280.85
Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.5, Memory (GB)=16.1, Evaluation protocol=zero-shot2026.0480.79
SARQC-GS(Identity)Model=LLaMA-30B, Precision=W4A16, Zero-shot=true2026.0580.79
BaselineBackbone=Qwen1.5-MoE-A2.7B, Type=–, Storage=28.67GB, Evaluation Protocol=zero-shot2026.0680.79
QuaRotModel=LLaDA-1.5, Quantization=W4A42026.0680.74
Llama3 8BCompression Ratio=Baseline2025.0980.7
Tensor RMS + CBit width (b)=3.002025.0580.7
Llama3 8B2026.0280.7
FP16Model=LLaMA-30B, Precision=W4A16, Zero-shot=true2026.0580.69
AWQModel=LLaMA-30B, Precision=W4A16, Zero-shot=true2026.0580.69
SARQC-GS(Saliency)Model=LLaMA-30B, Precision=W4A16, Zero-shot=true2026.0580.69
FP16Model=LLaMA-30B, Precision=W3A16, Zero-shot=true2026.0580.69
PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.5, Memory (GB)=16.1, Evaluation protocol=zero-shot2026.0480.63
Mashup Learning at initBackbone=Mistral-7B-Instruct-v0.2, Setup=Lots-of-LoRAs2026.0380.6
SARQC-GBS(Identity)Model=LLaMA-30B, Precision=W4A16, Zero-shot=true2026.0580.58
BF16Backbone=Llama-2-13B2026.0580.52
BF16Models=Qwen3-30B-A3B, Quantization Strategy=Embedding-wise, Quantization Bit-width=BF162026.0480.5
PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.375, Memory (GB)=15.3, Evaluation protocol=zero-shot2026.0480.47
BaselineModel=Qwen3-MoE, Weight Bits=16, Zero-shot=true2026.0480.47
Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.25, Memory (GB)=14.5, Evaluation protocol=zero-shot2026.0480.41
LLaMA-2 13BModel=LLaMA-2 13B2026.0380.4
Focus (LLaMA-2 13B)Backbone=LLaMA-2 13B, K=2, dg=162026.0380.4
BF16Models=DeepSeek-V2-Lite, Quantization Strategy=Embedding-wise, Quantization Bit-width=BF162026.0480.4
CodeQuantModel=DeepSeek-V2-Lite, Quantization Precision=A8W4, Quantization Granularity=Block-wise2026.0480.4
BaselineBackbone=Mixtral, Weight Bitwidth=162026.0480.36
GPTAQModel=LLaMA-30B, Precision=W4A16, Zero-shot=true2026.0580.25
HessianModel=Mixtral 8x7B, Avg. bits/exp.=2.5, Memory (GB)=17.0, Evaluation protocol=zero-shot2026.0480.21
Text-to-LoRABackbone=Mistral-7B-Instruct-v0.2, Setup=Lots-of-LoRAs2026.0380.2
Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.375, Memory (GB)=15.3, Evaluation protocol=zero-shot2026.0480.2
Baseline (Dense)Sparsity=0.0%, Zero-shot=true2026.0580.2
PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.25, Memory (GB)=14.5, Evaluation protocol=zero-shot2026.0480.14
Router norm + Max varModel=Mixtral 8x22B, Avg. bits/exp.=2.5, Memory (GB)=46.7, Evaluation protocol=zero-shot2026.0480.14
GPTQModel=LLaMA-30B, Precision=W4A16, Zero-shot=true2026.0580.14
SARQC-GBS(Saliency)Model=LLaMA-30B, Precision=W3A16, Zero-shot=true2026.0580.14
SARQC-GBS(Saliency)Model=LLaMA-7B, Precision=W4A16, Zero-shot=true2026.0580.03
SARQC-GBS(Identity)Model=LLaMA-30B, Precision=W3A16, Zero-shot=true2026.0580.03
STLBase Model=Qwen2.5-7B2026.0280
STL-augBase Model=Qwen2.5-7B2026.0280
SurgicalBase Model=Qwen2.5-7B2026.0280
Qwen 2.5 7BModel=Qwen 2.5 7B2026.0380
Focus (Qwen 2.5 7B)Backbone=Qwen 2.5 7B, K=4, dg=162026.0380
FVBackbone=Qwen2.5-7B2025.0980
SARQC-GBS(Saliency)Model=LLaMA-13B, Precision=W4A16, Zero-shot=true2026.0579.92
WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P25%, Storage=28.67GB, Evaluation Protocol=zero-shot2026.0679.87
CodeQuantModel=DeepSeek-V2-Lite, Quantization Precision=A8W4, Quantization Granularity=Embedding-wise2026.0479.8
BaselineModel=OLMoE, Weight Bits=16, Zero-shot=true2026.0479.76
BaselineBackbone=Qwen1.5-MoE, Weight Bitwidth=162026.0479.76
BF16Models=Phi-mini MoE-Instruct, Quantization Strategy=Embedding-wise, Quantization Bit-width=BF162026.0479.7
CodeQuantModel=Qwen3-30B-A3B, Quantization Precision=A8W4, Quantization Granularity=Embedding-wise2026.0479.7
PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Type=M25%, Storage=22.40GB, Evaluation Protocol=zero-shot2026.0679.7
FP16Model=LLaMA-13B, Precision=W4A16, Zero-shot=true2026.0579.65
FP16Model=LLaMA-13B, Precision=W3A16, Zero-shot=true2026.0579.65
CodeQuantModel=Phi-mini MoE-Instruct, Quantization Precision=A8W4, Quantization Granularity=Embedding-wise2026.0479.6
BaselineBackbone=Qwen3-MoE-30B-A3B, Type=–, Storage=61.06GB, Evaluation Protocol=zero-shot2026.0679.6
DuQuant++*Model=LLaMA3.1-8B-Instruct, Precision=MXFP42026.0479.5
PMQModel=Mixtral 8x22B, Avg. bits/exp.=2.5, Memory (GB)=46.7, Evaluation protocol=zero-shot2026.0479.49
CodeQuantModels=DeepSeek-V2-Lite, Quantization Strategy=Block-wise, Quantization Bit-width=A4W42026.0479.4
PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Type=M50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.0679.4
GPTQModel=LLaMA-13B, Precision=W4A16, Zero-shot=true2026.0579.38
SARQC-GBS(Identity)Model=LLaMA-13B, Precision=W4A16, Zero-shot=true2026.0579.38
WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=7.10GB, Evaluation Protocol=zero-shot2026.0679.38
OursBackbone=Qwen3-MoE-30B-A3B, Type=P50%, Storage=32.07GB, Evaluation Protocol=zero-shot2026.0679.33