Common Sense Reasoning on Winogrande
91.3AccuracySupervised SOTA
Evaluation Results
| Method | Links | |
|---|---|---|
| Supervised SOTAsupervised=true2022.03 | 91.3 | |
| Nemotron 4 340BParameters=340B2024.07 | 89.5 | |
| CALDERABackbone=LLaMa-3 8B, Rank=256, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.72024.05 | 89.19 | |
| CALDERABackbone=LLaMa-3 8B, Rank=128, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.52024.05 | 88.63 | |
| CALDERABackbone=LLaMa-3 8B, Rank=64, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.42024.05 | 88.4 | |
| CALDERABackbone=LLaMa-3 8B, Rank=64, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.42024.05 | 88.4 | |
| CALDERABackbone=LLaMa-3 8B, Rank=64, BQ=2, BL (= BR)=16, RHT FT=Yes, Avg Bits=2.42024.05 | 88.16 | |
| CALDERABackbone=LLaMa-3 8B, Rank=256, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.72024.05 | 88.08 | |
| CALDERABackbone=LLaMa-3 8B, Rank=128, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.52024.05 | 88 | |
| GPT-4Number of shots=5-shot2023.11 | 87.5 | |
| GPT-42024.07 | 87.5 | |
| Falcon-180BNumber of shots=5-shot2023.11 | 87.1 | |
| FalconModel Size=180B, Evaluation Protocol=5-shot2023.11 | 87.1 | |
| MixDoRABase Model=LLaMA-2 13B, Trainable Parameters=2.5%2024.04 | 86.5 | |
| MixLoRABase Model=LLaMA-2 13B, Trainable Parameters=2.5%2024.04 | 86.3 | |
| FlexoraFine-tuning Strategy=LoRA, Enhanced=Flexora2024.08 | 85.79 | |
| Flexora(w/ Base Model)Backbone=Meta-Llama-3-8B, Model variant=Base, Adaptation=Flexora2024.08 | 85.79 | |
| Flexora(w/ Instruct Model)Backbone=Meta-Llama-3-8B-Instruct, Model variant=Instruct, Adaptation=Flexora2024.08 | 85.67 | |
| CALDERABackbone=LLaMa-3 8B, Rank=64, BQ=2, BL (= BR)=16, RHT FT=No, Avg Bits=2.42024.05 | 85.56 | |
| CALDERABackbone=LLaMa-2 7B, Rank=256, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.72024.05 | 85.4 | |
| CALDERABackbone=LLaMa-2 7B, Rank=64, BQ=2, BL (= BR)=16, RHT FT=Yes, Avg Bits=2.42024.05 | 85.32 | |
| CALDERABackbone=LLaMa-2 7B, Rank=128, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.52024.05 | 85.32 | |
| Falcon-180Bprotocol=1-shot2023.11 | 85.1 | |
| CALDERABackbone=LLaMa-2 7B, Rank=128, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.52024.05 | 85 | |
| CALDERABackbone=LLaMa-2 7B, Rank=256, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.72024.05 | 84.93 | |
| Mixtral 8×22BParameters=8×22B2024.07 | 84.7 | |
| MixDoRABase Model=LLaMA-3 8B, Trainable Parameters=3.0%2024.04 | 84.2 | |
| CALDERABackbone=LLaMa-2 7B, Rank=64, BQ=2, BL (= BR)=16, RHT FT=No, Avg Bits=2.42024.05 | 84.06 | |
| CALDERABackbone=LLaMa-2 7B, Rank=64, BQ=2, BL (= BR)=4, RHT FT=No, Avg Bits=2.42024.05 | 84.06 | |
| PaLMprotocol=1-shot2023.11 | 83.7 | |
| DoRABase Model=LLaMA-3 8B, Trainable Parameters=2.6%2024.04 | 83.7 | |
| Llama 3 70BParameters=70B2024.07 | 83.5 | |
| CALDERABackbone=LLaMa-2 7B, Rank=64, BQ=2, BL (= BR)=4, RHT FT=Yes, Avg Bits=2.42024.05 | 83.42 | |
| MixDoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.04 | 83.4 | |
| Inflection-1Evaluation Protocol=5-shot2023.11 | 83.3 | |
| PaLM-2 Lprotocol=1-shot2023.11 | 83 | |
| DoRABase Model=LLaMA-2 13B, Trainable Parameters=2.4%2024.04 | 82.4 | |
| LoRAFine-tuning Strategy=LoRA2024.08 | 82.24 | |
| Llama 3 405BParameters=405B2024.07 | 82.2 | |
| MixLoRABase Model=LLaMA-3 8B, Trainable Parameters=3.0%2024.04 | 82.1 | |
| LoRABase Model=LLaMA-2 13B, Trainable Parameters=2.4%2024.04 | 81.9 | |
| FalconModel Size=40B, Evaluation Protocol=5-shot2023.11 | 81.8 | |
| FlexoraFine-tuning Strategy=Full Fine-Tuning, Enhanced=Flexora2024.08 | 81.73 | |
| GPT-3.5Number of shots=5-shot2023.11 | 81.6 | |
| FP16Model=Qwen2.5-14B2026.03 | 81.45 | |
| Instruct ModelBackbone=Meta-Llama-3-8B-Instruct, Model variant=Instruct2024.08 | 81.35 | |
| Full FTFine-tuning Strategy=Full Fine-Tuning2024.08 | 81.16 | |
| PaLMEvaluation Protocol=0-shot2023.11 | 81.1 | |
| FalconModel Size=180B, Evaluation Protocol=0-shot2023.11 | 80.3 | |
| LLaMA-2Model Size=70B, Evaluation Protocol=0-shot2023.11 | 80.2 | |
| PaLM-2 Mprotocol=1-shot2023.11 | 79.2 | |
| Mistral 7BParameters=7B2024.07 | 78.1 | |
| PaLM-2 Sprotocol=1-shot2023.11 | 77.9 | |
| FP16Backbone=Llama-3.1-8B2026.03 | 77.35 | |
| FP16 BaselineModel=LLaMA-2-70B, #Bits=W16A162024.02 | 76.95 | |
| MixLoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.04 | 76.8 | |
| LLaMA-2Model Size=34B, Evaluation Protocol=0-shot2023.11 | 76.7 | |
| FP16Model=Llama-2-13B2026.03 | 76.16 | |
| FP16Backbone=Qwen2.5-7B2026.03 | 76.01 | |
| FalconModel Size=40B, Evaluation Protocol=0-shot2023.11 | 76 | |
| Llama 3 8BParameters=8B2024.07 | 75.7 | |
| LoRABase Model=LLaMA-3 8B, Trainable Parameters=2.6%2024.04 | 75.3 | |
| Chinchillazero-shot=true2022.03 | 74.9 | |
| ChinchillaEvaluation Protocol=0-shot2023.11 | 74.9 | |
| Gemma 7BParameters=7B2024.07 | 74.7 | |
| NSDSModel=Qwen2.5-14B2026.03 | 74.28 | |
| NSDSModel=Llama-2-13B2026.03 | 73.96 | |
| ZDModel=Qwen2.5-14B2026.03 | 73.91 | |
| UnquantizedRank=0, Avg Bits=16, Backbone=LLaMa-3 8B, Evaluation Mode=Zero-shot2024.05 | 73.5 | |
| EWQModel=Llama-2-13B2026.03 | 73.43 | |
| DBLLMModel=LLaMA-2-70B, #Bits=W2A162024.02 | 73.32 | |
| KurtBoostModel=Llama-2-13B2026.03 | 73.12 | |
| KurtosisModel=Qwen2.5-14B2026.03 | 73.06 | |
| MSEModel=Qwen2.5-14B2026.03 | 73.05 | |
| MT-NLG 530Bzero-shot=true2022.03 | 73 | |
| MT-NLGEvaluation Protocol=0-shot2023.11 | 73 | |
| LLaMA-2Model Size=13B, Evaluation Protocol=0-shot2023.11 | 72.8 | |
| ZDModel=Llama-2-13B2026.03 | 72.69 | |
| FalconModel Size=7B, Evaluation Protocol=5-shot2023.11 | 72.6 | |
| EWQModel=Qwen2.5-14B2026.03 | 72.58 | |
| NSDSBackbone=Llama-3.1-8B2026.03 | 72.28 | |
| MSEModel=Llama-2-13B2026.03 | 71.63 | |
| CALDERABackbone=Mistral 7B, Rank=128, Avg Bits=2.192024.05 | 71.51 | |
| LLaMA-7BDistillation type=undistilled, Zero-shot=true2026.03 | 70.5 | |
| CALDERABackbone=Mistral 7B, Rank=256, Avg Bits=2.382024.05 | 70.24 | |
| GPT-3zero-shot=true2022.03 | 70.2 | |
| GPT-3Model Size=175B, Evaluation Protocol=Zero-shot2022.10 | 70.2 | |
| GPT-3Evaluation Protocol=0-shot2023.11 | 70.2 | |
| Gopherzero-shot=true2022.03 | 70.1 | |
| GopherEvaluation Protocol=0-shot2023.11 | 70.1 | |
| ARMADALoss Function=L_euclid, Teacher Model=Stable Diffusion, Student Model=LLaMA-7B, Zero-shot=true2026.03 | 70 | |
| GPT-3Model Variant=DaVinci, Zero-shot=true2022.04 | 69.9 | |
| CALDERARank=256, BL=4, Avg Bits=2.4, Backbone=LLaMa-3 8B, BQ=2 bits, Evaluation Mode=Zero-shot2024.05 | 69.7 | |
| KurtBoostBackbone=Llama-3.1-8B2026.03 | 69.69 | |
| FP16 BaselineModel=LLaMA-2-13B, #Bits=W16A162024.02 | 69.61 | |
| ARMADALoss Function=L_elementwise, Teacher Model=Stable Diffusion, Student Model=LLaMA-7B, Zero-shot=true2026.03 | 69.6 | |
| ARMADALoss Function=L_cosine, Teacher Model=Stable Diffusion, Student Model=LLaMA-7B, Zero-shot=true2026.03 | 69.5 | |
| QuIP#Backbone=Mistral 7B, Rank=0, Avg Bits=22024.05 | 69.3 | |
| LLaMA-2Model Size=7B, Evaluation Protocol=0-shot2023.11 | 69.2 | |
| FairSeqNumber of Parameters=13B, Evaluation Protocol=5-shot2022.04 | 69 |