Physical Interaction Question Answering on PIQA
94.9AccuracyHuman Performance
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Human Performance2024.01 | 94.9 | — | |
| Dense ModelBase Model=Mistral-7B, Fine-tuning Protocol=FFT, Sparsity=0%2026.01 | 90.3 | — | |
| GradPrunerBase Model=Mistral-7B, Fine-tuning Protocol=FFT, Sparsity=40%2026.01 | 89.7 | — | |
| Dense ModelBase Model=Mistral-7B, Fine-tuning Protocol=LoRA, Sparsity=0%2026.01 | 89.6 | — | |
| GradPrunerBase Model=Mistral-7B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 89.6 | — | |
| UNIFIEDQAModel Architecture=T52020.05 | 89.5 | — | |
| Dense ModelBase Model=Llama3.1-8B, Fine-tuning Protocol=LoRA, Sparsity=0%2026.01 | 89.3 | — | |
| GradPrunerBase Model=Llama3.1-8B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 89.1 | — | |
| T5Model Architecture=T52020.05 | 88.9 | — | |
| VERA-T5 (Multitask Supervised)Evaluation Protocol=Supervised2024.01 | 88.5 | — | |
| DoRABase Model=LLaMA-2 13B, Trainable Parameters=2.4%2024.04 | 87.8 | — | |
| MixLoRABase Model=LLaMA-3 8B, Trainable Parameters=3.0%2024.04 | 87.6 | — | |
| GradPrunerBase Model=Llama3.1-8B, Fine-tuning Protocol=FFT, Sparsity=40%2026.01 | 87.6 | — | |
| SATBase Model=Mistral-7B, Fine-tuning Protocol=FFT, Sparsity=40%2026.01 | 87 | — | |
| MixDoRABase Model=LLaMA-2 13B, Trainable Parameters=2.5%2024.04 | 86.9 | — | |
| MixLoRABase Model=LLaMA-2 13B, Trainable Parameters=2.5%2024.04 | 86.8 | — | |
| LoRABase Model=LLaMA-2 13B, Trainable Parameters=2.4%2024.04 | 86.7 | — | |
| Dense ModelBase Model=Llama3.1-8B, Fine-tuning Protocol=FFT, Sparsity=0%2026.01 | 86.7 | — | |
| Falcon-180Bprotocol=1-shot2023.11 | 86.1 | — | |
| FT(Llama3.2)Base Model=Llama3.2-3B, Fine-tuning Protocol=FFT, Sparsity=0%2026.01 | 86.1 | — | |
| APTBase Model=Llama3.1-8B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 85.9 | — | |
| FT(Llama3.2)Base Model=Llama3.2-3B, Fine-tuning Protocol=LoRA, Sparsity=0%2026.01 | 85.9 | — | |
| LacoBase Model=Mistral-7B, Fine-tuning Protocol=FFT, Sparsity=40%2026.01 | 85.7 | — | |
| APTBase Model=Mistral-7B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 85.4 | — | |
| PaLM-2 Lprotocol=1-shot2023.11 | 85 | — | |
| MINITRONBase Model=Mistral-7B, Fine-tuning Protocol=FFT, Sparsity=40%2026.01 | 84.6 | — | |
| LacoBase Model=Mistral-7B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 84.6 | — | |
| DeBERTa-v3-L (Supervised)Evaluation Protocol=Supervised2024.01 | 84.5 | — | |
| SATBase Model=Llama3.1-8B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 84.4 | — | |
| SATBase Model=Mistral-7B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 84.3 | — | |
| PaLMprotocol=1-shot2023.11 | 83.9 | — | |
| LLMPrunerBase Model=Mistral-7B, Fine-tuning Protocol=FFT, Sparsity=40%2026.01 | 83.9 | — | |
| LacoBase Model=Llama3.1-8B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 83.7 | — | |
| MINITRONBase Model=Mistral-7B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 83.5 | — | |
| MixDoRABase Model=LLaMA-3 8B, Trainable Parameters=3.0%2024.04 | 83.4 | — | |
| SATBase Model=Llama3.1-8B, Fine-tuning Protocol=FFT, Sparsity=40%2026.01 | 83.3 | — | |
| PaLM-2 Mprotocol=1-shot2023.11 | 83.2 | — | |
| MixLoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.04 | 83.2 | — | |
| Mistral 7BModality=Pretrained2023.10 | 83 | — | |
| Mistral-v0.1Parameters=7B, Evaluation Protocol=Zero-shot2024.01 | 83 | — | |
| DoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.04 | 82.7 | — | |
| LLMPrunerBase Model=Mistral-7B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 82.7 | — | |
| LLMPrunerBase Model=Llama3.1-8B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 82.6 | — | |
| DoRABase Model=LLaMA-3 8B, Trainable Parameters=2.6%2024.04 | 82.3 | — | |
| PaLM-2 Sprotocol=1-shot2023.11 | 82.2 | — | |
| MixDoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.04 | 82.2 | — | |
| MINITRONBase Model=Llama3.1-8B, Fine-tuning Protocol=LoRA, Sparsity=40%2026.01 | 82.2 | — | |
| LoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.04 | 82.1 | — | |
| MixDoRA# Params (%)=4.3%, ST/MT=ST, Base Model=Gemma-2B2024.04 | 81.8 | — | |
| MixDoRABase Model=Gemma 2B, Trainable Parameters=4.3%2024.04 | 81.8 | — | |
| ChatGPT + Self-consistent chain-of-thoughtVersion=gpt-3.5-turbo, Evaluation Protocol=Zero-shot2024.01 | 81.7 | — | |
| DoRA# Params (%)=3.2%, ST/MT=ST, Base Model=Gemma-2B2024.04 | 81.6 | — | |
| DoRABase Model=Gemma 2B, Trainable Parameters=3.2%2024.04 | 81.6 | — | |
| BF16-BaselineEff. Bits=16, Model=Llama-3.1-8B-Instruct2026.06 | 81.34 | — | |
| HARPBackbone=Llama 2 70B, Target Bits=4, BPP=4.04, Evaluation Protocol=Zero-shot, Evaluation Harness=lm_eval2026.05 | 81.2 | — | |
| DoRA# Params (%)=3.2%, ST/MT=MT, Base Model=Gemma-2B2024.04 | 81.1 | — | |
| MixLoRA# Params (%)=4.3%, ST/MT=ST, Base Model=Gemma-2B2024.04 | 81.1 | — | |
| MixLoRABase Model=Gemma 2B, Trainable Parameters=4.3%2024.04 | 81.1 | — | |
| FP16Backbone=Llama 2 70B, Target Bits=16, BPP=16.00, Evaluation Protocol=Zero-shot, Evaluation Harness=lm_eval2026.05 | 81.1 | — | |
| MatGPTQEff. Bits=4.00, Model=Llama-3.1-8B-Instruct2026.06 | 80.96 | — | |
| LoRA# Params (%)=3.2%, ST/MT=ST, Base Model=Gemma-2B2024.04 | 80.9 | — | |
| LoRABase Model=Gemma 2B, Trainable Parameters=3.2%2024.04 | 80.9 | — | |
| LacoBase Model=Llama3.1-8B, Fine-tuning Protocol=FFT, Sparsity=40%2026.01 | 80.9 | — | |
| LLaMA 2 13BModality=Pretrained2023.10 | 80.8 | — | |
| LoRABase Model=LLaMA-3 8B, Trainable Parameters=2.6%2024.04 | 80.7 | — | |
| QuIP# (RHT)Backbone=Llama 2 70B, Target Bits=3, BPP=3.00, Evaluation Protocol=Zero-shot, Evaluation Harness=lm_eval2026.05 | 80.7 | — | |
| HARPBackbone=Llama 2 70B, Target Bits=3, BPP=3.04, Evaluation Protocol=Zero-shot, Evaluation Harness=lm_eval2026.05 | 80.6 | — | |
| QuIP# (RHT)Backbone=Llama 2 70B, Target Bits=4, BPP=4.00, Evaluation Protocol=Zero-shot, Evaluation Harness=lm_eval2026.05 | 80.6 | — | |
| LLMPrunerBase Model=Llama3.1-8B, Fine-tuning Protocol=FFT, Sparsity=40%2026.01 | 80.5 | — | |
| GRINQH-4b-GPTQEff. Bits=3.01, Model=Llama-3.1-8B-Instruct2026.06 | 80.47 | — | |
| DeBERTa-v3-L (CANDLE Distilled)CSKB=CANDLE, Evaluation Protocol=Zero-shot2024.01 | 80.3 | — | |
| MINITRONBase Model=Llama3.1-8B, Fine-tuning Protocol=FFT, Sparsity=40%2026.01 | 80.3 | — | |
| GRINQH-8b-RTNEff. Bits=3.96, Model=Llama-3.1-8B-Instruct2026.06 | 80.3 | — | |
| GRINQH-6b-RTNEff. Bits=3.881, Model=Llama-3.1-8B-Instruct2026.06 | 80.25 | — | |
| GRINQH-8b-RTNEff. Bits=2.95, Model=Llama-3.1-8B-Instruct2026.06 | 80.25 | — | |
| LLAMA2Parameters=13B, Evaluation Protocol=Zero-shot2024.01 | 80.2 | — | |
| OLMoE-1B-7BModel Type=MoE, Active Parameters=1B, Total Parameters=7B2026.02 | 80.1 | — | |
| SpB1.0-7BModel Size=7B2026.04 | 80.03 | — | |
| DeepSeek V2 LiteStrategy=R&Q2026.02 | 79.9 | 3.2925 | |
| HARPBackbone=Llama 2 70B, Target Bits=2, BPP=2.04, Evaluation Protocol=Zero-shot, Evaluation Harness=lm_eval2026.05 | 79.9 | — | |
| DeepSeek V2 LiteStrategy=Raw2026.02 | 79.8 | 4.3504 | |
| MatGPTQ-EP-Mix’n’MatchEff. Bits=4.00, Model=Llama-3.1-8B-Instruct2026.06 | 79.76 | — | |
| GEMMA-7BState=Dense2025.06 | 79.71 | — | |
| Qwen2.5-7BModel Type=Dense, Active Parameters=7B, Total Parameters=7B2026.02 | 79.7 | — | |
| OLMo-7BModel Type=Dense, Active Parameters=7B, Total Parameters=7B2026.02 | 79.6 | — | |
| GRINQH-6b-RTNEff. Bits=2.98, Model=Llama-3.1-8B-Instruct2026.06 | 79.54 | — | |
| QuIP# (RHT)Backbone=Llama 2 70B, Target Bits=2, BPP=2.00, Evaluation Protocol=Zero-shot, Evaluation Harness=lm_eval2026.05 | 79.5 | — | |
| RoBERTaModel Architecture=RoBERTa2020.05 | 79.4 | — | |
| DPModel Scale=Gemma 9B, Method=DP2026.04 | 79.3 | — | |
| DEEPSEEK-7BState=Dense2025.06 | 79.27 | — | |
| ChatGPT + Chain-of-thoughtVersion=gpt-3.5-turbo, Evaluation Protocol=Zero-shot2024.01 | 79.2 | — | |
| ROBERTa-L (Supervised)Evaluation Protocol=Supervised2024.01 | 79.2 | — | |
| G-AdamWShots=3, Optimizer=Global AdamW, Model=LLaMA 7B, Instruction Finetuning=true2024.03 | 79.18 | — | |
| G-LionShots=3, Optimizer=Global Lion, Model=LLaMA 7B, Instruction Finetuning=true2024.03 | 79.18 | — | |
| MixDoRA# Params (%)=4.3%, ST/MT=MT, Base Model=Gemma-2B2024.04 | 79.1 | — | |
| DeepSeek-MoEStrategy=R&Q2026.02 | 79.1 | 3.0585 | |
| Gemma3-4BModel Size=4B2026.04 | 79.05 | — | |
| DeBERTa-v3-L (MR)CSKB=ATM10X, Evaluation Protocol=Zero-shot2024.01 | 79 | — | |
| FP16BIT=16, Evaluation Protocol=Few-shot2026.07 | 78.94 | — | |
| Distributed Lion-MaVoShots=3, Optimizer=D-Lion (MaVo), Model=LLaMA 7B, Instruction Finetuning=true2024.03 | 78.92 | — |