List your company
ResearchFormerly WizWand

Commonsense Reasoning on Winogrande (Accuracy and Standard Deviation)

74.9AccuracyFP
45.5253.147560.77568.4025Apr 17, 2025Jun 24, 2025Sep 1, 2025Nov 9, 2025Jan 16, 2026Mar 26, 2026Jun 3, 2026
Updated 4mo ago

Evaluation results

MethodLinks
FPModel=LLADA, Precision=Full Precision2026.0674.9——
FPModel=LLADA-1.5, Precision=Full Precision2026.0674.8——
FPModel=DREAM, Precision=Full Precision2026.0674.5——
STaR-QuantModel=LLADA-1.5, Quantization=W8A82026.0673.98——
Llama-3.1-8BArchitecture=Transformer, Tokens (B)=15000, Evaluation Protocol=zero-shot2025.0573.9——
RecurrentGemma-9BArchitecture=Subquadratic (Pretrained), Tokens (B)=2000, Evaluation Protocol=zero-shot2025.0573.7——
LoLCATs-8BArchitecture=Subquadratic (Distilled), Tokens (B)=0.04, Evaluation Protocol=zero-shot2025.0573.6——
LoLA-8B (ours)Architecture=Subquadratic (Distilled), Tokens (B)=0.04, Evaluation Protocol=zero-shot, η=64, λ=642025.0573.6——
STaR-QuantModel=DREAM, Quantization=W8A82026.0673.5——
Llamba-8BArchitecture=Subquadratic (Distilled), Tokens (B)=12, Evaluation Protocol=zero-shot2025.0573.3——
Griffin 7BArchitecture=Subquadratic (Pretrained), Tokens (B)=300, Evaluation Protocol=zero-shot2025.0572.6——
Mamba2-8BArchitecture=Subquadratic (Pretrained), Tokens (B)=3500, Evaluation Protocol=zero-shot2025.0571.6——
RWKV-6 (W2.1) 7BArchitecture=Subquadratic (Pretrained), Tokens (B)=1420, Evaluation Protocol=zero-shot2025.0570——
Hawk 7BArchitecture=Subquadratic (Pretrained), Tokens (B)=300, Evaluation Protocol=zero-shot2025.0569.9——
SparseGPTBackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.1069.77——
L-ADMMBackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.1069.3——
SparseLLMBackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.1069.3——
Falcon3-Mamba-7BArchitecture=Subquadratic (Pretrained), Tokens (B)=7300, Evaluation Protocol=zero-shot2025.0569.1——
DenseBackbone=Llama-2-7B, Sparsity=0%, Evaluation Protocol=Zero-shot2025.1069.06——
ALPSBackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.1068.98——
STaR-QuantModel=LLADA, Quantization=W8A82026.0668.72——
Wanda + GRAILSparsity=20%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0268.43——
Mamba-8BArchitecture=Subquadratic (Pretrained), Tokens (B)=1100, Evaluation Protocol=zero-shot2025.0568.3——
Wanda++Sparsity=20%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0268.27——
SlimGPT + GRAILSparsity=20%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0268.19——
Few-shotModel=Qwen3-8B, Training Size=1002026.0267.840.13—
Wanda++ + GRAILSparsity=20%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0267.72——
WandaBackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.1067.17——
ZipLMSparsity=20%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0266.85——
SAFEBackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.1066.77——
L-ADMMBackbone=Llama-2-7B, Sparsity=60%, Evaluation Protocol=Zero-shot2025.1066.61——
ELSABackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.1066.54——
FLAPSparsity=20%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0266.38——
ALPSBackbone=Llama-2-7B, Sparsity=60%, Evaluation Protocol=Zero-shot2025.1066.29——
SparseLLMBackbone=Llama-2-7B, Sparsity=60%, Evaluation Protocol=Zero-shot2025.1066.14——
FLAP + GRAILSparsity=20%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0266.06——
SparseGPTBackbone=Llama-2-7B, Sparsity=60%, Evaluation Protocol=Zero-shot2025.1065.51——
WandaSparsity=20%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0264.17——
WandaBackbone=Llama-2-7B, Sparsity=60%, Evaluation Protocol=Zero-shot2025.1064.01——
CLIMBSize=950M2025.0463.54——
ELSABackbone=Llama-2-7B, Sparsity=60%, Evaluation Protocol=Zero-shot2025.1063.38——
MagnitudeBackbone=Llama-2-7B, Sparsity=50%, Evaluation Protocol=Zero-shot2025.1063.22——
SAFEBackbone=Llama-2-7B, Sparsity=60%, Evaluation Protocol=Zero-shot2025.1062.98——
ALPSBackbone=Llama-2-7B, Sparsity=70%, Evaluation Protocol=Zero-shot2025.1061.72——
AMD-OLMoSize=1.2B2025.0461.64——
L-ADMMBackbone=Llama-2-7B, Sparsity=70%, Evaluation Protocol=Zero-shot2025.1060.77——
Llama-3.2Size=1.2B2025.0460.69——
FLAPSparsity=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0259.83——
FLAP + GRAILSparsity=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0259.83——
CCGQAParameters=1B, Training Tokens=300B, Architecture=Dense, Evaluation Protocol=Zero-shot, KV-cache compression=1/42025.1059.7——
TinyLlamaSize=1.1B2025.0459.12——
ELSABackbone=Llama-2-7B, Sparsity=70%, Evaluation Protocol=Zero-shot2025.1058.64——
SlimGPT + GRAILSparsity=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0258.64——
Mamba2-Llama3-8B (L3.1-Instr.)Architecture=Subquadratic (Distilled), Tokens (B)=20, Evaluation Protocol=zero-shot2025.0558.6——
CLIMBSize=350M2025.0457.93——
SparseGPTBackbone=Llama-2-7B, Sparsity=70%, Evaluation Protocol=Zero-shot2025.1057.7——
MLAParameters=1B, Training Tokens=300B, Architecture=Dense, Evaluation Protocol=Zero-shot, KV-cache compression=1/42025.1057.6——
SparseLLMBackbone=Llama-2-7B, Sparsity=70%, Evaluation Protocol=Zero-shot2025.1057.46——
GQAParameters=1B, Training Tokens=300B, Architecture=Dense, Evaluation Protocol=Zero-shot, KV-cache compression=1/42025.1056.9——
MHAParameters=1B, Training Tokens=300B, Architecture=Dense, Evaluation Protocol=Zero-shot, KV-cache compression=1/12025.1056.8——
SmolLMSize=360M2025.0456.75——
CCAParameters=1B, Training Tokens=300B, Architecture=Dense, Evaluation Protocol=Zero-shot, KV-cache compression=1/42025.1056.7——
Qwen2.5Size=490M2025.0456.59——
ZipLMSparsity=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0255.64——
Wanda + GRAILSparsity=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0254.46——
Hedgehog-8B (Llama-3)Architecture=Subquadratic (Distilled), Tokens (B)=0.04, Evaluation Protocol=zero-shot2025.0554.3——
SlimGPTSparsity=20%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0253.59——
ELSABackbone=Llama-2-7B, Sparsity=80%, Evaluation Protocol=Zero-shot2025.1053.2——
MagnitudeBackbone=Llama-2-7B, Sparsity=60%, Evaluation Protocol=Zero-shot2025.1053.12——
SAFEBackbone=Llama-2-7B, Sparsity=70%, Evaluation Protocol=Zero-shot2025.1053.12——
SlimGPTSparsity=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0252.96——
Wanda++ + GRAILSparsity=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0252.8——
ROASTModel=Gemma3-1B, Training Size=1000, Aggregation=no-group2026.0252.560.01—
ROASTModel=Qwen3-0.6B, Training Size=1000, Aggregation=group2026.0252.360—
ELSABackbone=Llama-2-7B, Sparsity=90%, Evaluation Protocol=Zero-shot2025.1051.7——
MagnitudeBackbone=Llama-2-7B, Sparsity=80%, Evaluation Protocol=Zero-shot2025.1051.62——
L-ADMMBackbone=Llama-2-7B, Sparsity=80%, Evaluation Protocol=Zero-shot2025.1051.22——
ALPSBackbone=Llama-2-7B, Sparsity=80%, Evaluation Protocol=Zero-shot2025.1050.91——
SparseGPTBackbone=Llama-2-7B, Sparsity=90%, Evaluation Protocol=Zero-shot2025.1050.59——
MagnitudeBackbone=Llama-2-7B, Sparsity=90%, Evaluation Protocol=Zero-shot2025.1050.12——
Wanda++Sparsity=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0250.12——
BaselineModel=Qwen3-0.6B2026.0249.80.59—
SAFEBackbone=Llama-2-7B, Sparsity=80%, Evaluation Protocol=Zero-shot2025.1049.8——
WandaBackbone=Llama-2-7B, Sparsity=90%, Evaluation Protocol=Zero-shot2025.1049.33——
MagnitudeBackbone=Llama-2-7B, Sparsity=70%, Evaluation Protocol=Zero-shot2025.1049.25——
WandaBackbone=Llama-2-7B, Sparsity=80%, Evaluation Protocol=Zero-shot2025.1049.25——
SparseGPTBackbone=Llama-2-7B, Sparsity=80%, Evaluation Protocol=Zero-shot2025.1049.01——
WandaBackbone=Llama-2-7B, Sparsity=70%, Evaluation Protocol=Zero-shot2025.1048.86——
SparseLLMBackbone=Llama-2-7B, Sparsity=80%, Evaluation Protocol=Zero-shot2025.1048.86——
WandaSparsity=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot, Calibration Dataset=C42026.0248.38——
SAFEBackbone=Llama-2-7B, Sparsity=90%, Evaluation Protocol=Zero-shot2025.1047.83——
L-ADMMBackbone=Llama-2-7B, Sparsity=90%, Evaluation Protocol=Zero-shot2025.1047.51——
SparseLLMBackbone=Llama-2-7B, Sparsity=90%, Evaluation Protocol=Zero-shot2025.1046.96——
ALPSBackbone=Llama-2-7B, Sparsity=90%, Evaluation Protocol=Zero-shot2025.1046.65——
Acc (Ours)Pruning Method=Accuracy-based relevance score, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance2026.05——60.14
BaseModel=LLaMA-3.1-8B, Compression=0%, Mode=Zero-shot2025.10——73.56
BaselineModel=Qwen2.52026.04——72.3
BaselineModel=LLaMA-3.12026.04——73.8
BaselineModel=Qwen32026.04——68.7
BaselineBackbone=Qwen1.5-MoE-A2.7B, Type=–, Storage=28.67GB, Evaluation Protocol=zero-shot2026.06——69.77