List your company
ResearchFormerly WizWand

Common Sense Reasoning on Winogrande

91.3AccuracySupervised SOTA
68.10874.12980.1586.171Mar 29, 2022Nov 25, 2022Jul 25, 2023Mar 23, 2024Nov 19, 2024Jul 19, 2025Mar 18, 2026
Updated 7mo ago

Evaluation results

MethodLinks
Supervised SOTAsupervised=true2022.0391.3
Nemotron 4 340BParameters=340B2024.0789.5
CALDERABackbone=LLaMa-3 8B, Rank=256, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.72024.0589.19
CALDERABackbone=LLaMa-3 8B, Rank=128, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.52024.0588.63
CALDERABackbone=LLaMa-3 8B, Rank=64, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.42024.0588.4
CALDERABackbone=LLaMa-3 8B, Rank=64, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.42024.0588.4
CALDERABackbone=LLaMa-3 8B, Rank=64, BQ=2, BL (= BR)=16, RHT FT=Yes, Avg Bits=2.42024.0588.16
CALDERABackbone=LLaMa-3 8B, Rank=256, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.72024.0588.08
CALDERABackbone=LLaMa-3 8B, Rank=128, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.52024.0588
GPT-4Number of shots=5-shot2023.1187.5
GPT-42024.0787.5
Falcon-180BNumber of shots=5-shot2023.1187.1
FalconModel Size=180B, Evaluation Protocol=5-shot2023.1187.1
MixDoRABase Model=LLaMA-2 13B, Trainable Parameters=2.5%2024.0486.5
MixLoRABase Model=LLaMA-2 13B, Trainable Parameters=2.5%2024.0486.3
FlexoraFine-tuning Strategy=LoRA, Enhanced=Flexora2024.0885.79
Flexora(w/ Base Model)Backbone=Meta-Llama-3-8B, Model variant=Base, Adaptation=Flexora2024.0885.79
Flexora(w/ Instruct Model)Backbone=Meta-Llama-3-8B-Instruct, Model variant=Instruct, Adaptation=Flexora2024.0885.67
CALDERABackbone=LLaMa-3 8B, Rank=64, BQ=2, BL (= BR)=16, RHT FT=No, Avg Bits=2.42024.0585.56
CALDERABackbone=LLaMa-2 7B, Rank=256, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.72024.0585.4
CALDERABackbone=LLaMa-2 7B, Rank=64, BQ=2, BL (= BR)=16, RHT FT=Yes, Avg Bits=2.42024.0585.32
CALDERABackbone=LLaMa-2 7B, Rank=128, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.52024.0585.32
Falcon-180Bprotocol=1-shot2023.1185.1
CALDERABackbone=LLaMa-2 7B, Rank=128, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.52024.0585
CALDERABackbone=LLaMa-2 7B, Rank=256, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.72024.0584.93
Mixtral 8×22BParameters=8×22B2024.0784.7
MixDoRABase Model=LLaMA-3 8B, Trainable Parameters=3.0%2024.0484.2
CALDERABackbone=LLaMa-2 7B, Rank=64, BQ=2, BL (= BR)=16, RHT FT=No, Avg Bits=2.42024.0584.06
CALDERABackbone=LLaMa-2 7B, Rank=64, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.42024.0584.06
PaLMprotocol=1-shot2023.1183.7
DoRABase Model=LLaMA-3 8B, Trainable Parameters=2.6%2024.0483.7
Llama 3 70BParameters=70B2024.0783.5
CALDERABackbone=LLaMa-2 7B, Rank=64, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.42024.0583.42
MixDoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.0483.4
Inflection-1Evaluation Protocol=5-shot2023.1183.3
PaLM-2 Lprotocol=1-shot2023.1183
DoRABase Model=LLaMA-2 13B, Trainable Parameters=2.4%2024.0482.4
LoRAFine-tuning Strategy=LoRA2024.0882.24
Llama 3 405BParameters=405B2024.0782.2
MixLoRABase Model=LLaMA-3 8B, Trainable Parameters=3.0%2024.0482.1
LoRABase Model=LLaMA-2 13B, Trainable Parameters=2.4%2024.0481.9
FalconModel Size=40B, Evaluation Protocol=5-shot2023.1181.8
FlexoraFine-tuning Strategy=Full Fine-Tuning, Enhanced=Flexora2024.0881.73
GPT-3.5Number of shots=5-shot2023.1181.6
FP16Model=Qwen2.5-14B2026.0381.45
Instruct ModelBackbone=Meta-Llama-3-8B-Instruct, Model variant=Instruct2024.0881.35
Full FTFine-tuning Strategy=Full Fine-Tuning2024.0881.16
PaLMEvaluation Protocol=0-shot2023.1181.1
FalconModel Size=180B, Evaluation Protocol=0-shot2023.1180.3
LLaMA-2Model Size=70B, Evaluation Protocol=0-shot2023.1180.2
PaLM-2 Mprotocol=1-shot2023.1179.2
Mistral 7BParameters=7B2024.0778.1
PaLM-2 Sprotocol=1-shot2023.1177.9
FP16Backbone=Llama-3.1-8B2026.0377.35
FP16 BaselineModel=LLaMA-2-70B, #Bits=W16A162024.0276.95
MixLoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.0476.8
LLaMA-2Model Size=34B, Evaluation Protocol=0-shot2023.1176.7
FP16Model=Llama-2-13B2026.0376.16
FP16Backbone=Qwen2.5-7B2026.0376.01
FalconModel Size=40B, Evaluation Protocol=0-shot2023.1176
Llama 3 8BParameters=8B2024.0775.7
LoRABase Model=LLaMA-3 8B, Trainable Parameters=2.6%2024.0475.3
Chinchillazero-shot=true2022.0374.9
ChinchillaEvaluation Protocol=0-shot2023.1174.9
Gemma 7BParameters=7B2024.0774.7
NSDSModel=Qwen2.5-14B2026.0374.28
NSDSModel=Llama-2-13B2026.0373.96
ZDModel=Qwen2.5-14B2026.0373.91
UnquantizedRank=0, Avg Bits=16, Backbone=LLaMa-3 8B, Evaluation Mode=Zero-shot2024.0573.5
EWQModel=Llama-2-13B2026.0373.43
DBLLMModel=LLaMA-2-70B, #Bits=W2A162024.0273.32
KurtBoostModel=Llama-2-13B2026.0373.12
KurtosisModel=Qwen2.5-14B2026.0373.06
MSEModel=Qwen2.5-14B2026.0373.05
MT-NLG 530Bzero-shot=true2022.0373
MT-NLGEvaluation Protocol=0-shot2023.1173
LLaMA-2Model Size=13B, Evaluation Protocol=0-shot2023.1172.8
ZDModel=Llama-2-13B2026.0372.69
FalconModel Size=7B, Evaluation Protocol=5-shot2023.1172.6
EWQModel=Qwen2.5-14B2026.0372.58
NSDSBackbone=Llama-3.1-8B2026.0372.28
MSEModel=Llama-2-13B2026.0371.63
CALDERABackbone=Mistral 7B, Rank=128, Avg Bits=2.192024.0571.51
LLaMA-7BDistillation type=undistilled, Zero-shot=true2026.0370.5
CALDERABackbone=Mistral 7B, Rank=256, Avg Bits=2.382024.0570.24
GPT-3zero-shot=true2022.0370.2
GPT-3Model Size=175B, Evaluation Protocol=Zero-shot2022.1070.2
GPT-3Evaluation Protocol=0-shot2023.1170.2
Gopherzero-shot=true2022.0370.1
GopherEvaluation Protocol=0-shot2023.1170.1
ARMADALoss Function=L_euclid, Teacher Model=Stable Diffusion, Student Model=LLaMA-7B, Zero-shot=true2026.0370
GPT-3Model Variant=DaVinci, Zero-shot=true2022.0469.9
CALDERARank=256, BL=4, Avg Bits=2.4, Backbone=LLaMa-3 8B, BQ=2 bits, Evaluation Mode=Zero-shot2024.0569.7
KurtBoostBackbone=Llama-3.1-8B2026.0369.69
FP16 BaselineModel=LLaMA-2-13B, #Bits=W16A162024.0269.61
ARMADALoss Function=L_elementwise, Teacher Model=Stable Diffusion, Student Model=LLaMA-7B, Zero-shot=true2026.0369.6
ARMADALoss Function=L_cosine, Teacher Model=Stable Diffusion, Student Model=LLaMA-7B, Zero-shot=true2026.0369.5
QuIP#Backbone=Mistral 7B, Rank=0, Avg Bits=22024.0569.3
LLaMA-2Model Size=7B, Evaluation Protocol=0-shot2023.1169.2
FairSeqNumber of Parameters=13B, Evaluation Protocol=5-shot2022.0469