Physical Commonsense Reasoning on PIQA (val)
83AccuracyMistral-v0.1
Evaluation Results
| Method | Links | |
|---|---|---|
| Mistral-v0.1Zero-shot=true, Model Scale=7B2024.01 | 83 | |
| ChatGPT + Self-consistent chain-of-thoughtZero-shot=true, Variant=gpt-3.5-turbo, Prompting Strategy=Self-consistent chain-of-thought2024.01 | 81.7 | |
| KVQuant-4b-1%Backbone=Mistral-7b, Bits Per FPN=4.32-4.352024.05 | 80.74 | |
| FP16Backbone=Mistral-7b, Bits Per FPN=162024.05 | 80.58 | |
| KVQuant-4bBackbone=Mistral-7b, Bits Per FPN=4.00-4.022024.05 | 80.58 | |
| CQ-2c8bBackbone=Mistral-7b, Bits Per FPN=4.002024.05 | 80.52 | |
| DeBERTa-v3-L (CANDLE Distilled)Zero-shot=true, CSKB=CANDLE2024.01 | 80.3 | |
| LLAMA2Zero-shot=true, Model Scale=13B2024.01 | 80.2 | |
| LLM-Pruner+FinetuneBackbone=Llama-2-13b, Param Ratio=0.90, Fine-tuning=Alpaca2025.12 | 79.7 | |
| KVQuant-2b-1%Backbone=Mistral-7b, Bits Per FPN=2.32-2.352024.05 | 79.65 | |
| CQ-4c8bBackbone=Mistral-7b, Bits Per FPN=2.002024.05 | 79.49 | |
| LLM-PrunerBackbone=Llama-2-13b, Param Ratio=0.90, Fine-tuning=None2025.12 | 79.4 | |
| KVQuant-4b-1%Backbone=LLaMA-2-13b, Bits Per FPN=4.32-4.352024.05 | 79.27 | |
| KVQuant-4bBackbone=LLaMA-13b, Bits Per FPN=4.00-4.022024.05 | 79.22 | |
| ChatGPT + Chain-of-thoughtZero-shot=true, Variant=gpt-3.5-turbo, Prompting Strategy=Chain-of-thought2024.01 | 79.2 | |
| LLM-Pruner+FinetuneBackbone=Llama-2-13b, Param Ratio=0.80, Fine-tuning=Alpaca2025.12 | 79.2 | |
| FP16Backbone=LLaMA-13b, Bits Per FPN=162024.05 | 79.16 | |
| FP16Backbone=LLaMA-2-13b, Bits Per FPN=162024.05 | 79.16 | |
| KVQuant-4b-1%Backbone=LLaMA-13b, Bits Per FPN=4.32-4.352024.05 | 79.16 | |
| CQ-2c8bBackbone=LLaMA-13b, Bits Per FPN=4.002024.05 | 79.11 | |
| BaselineBackbone=Llama-2-13b, Param Ratio=1.00, Fine-tuning=None2025.12 | 79.1 | |
| DeBERTa-v3-L (MR)Zero-shot=true, CSKB=ATM10X2024.01 | 79 | |
| KVQuant-4bBackbone=LLaMA-2-13b, Bits Per FPN=4.00-4.022024.05 | 78.94 | |
| LLAMA2Zero-shot=true, Model Scale=7B2024.01 | 78.8 | |
| FP16Backbone=LLaMA-7b, Bits Per FPN=162024.05 | 78.67 | |
| KVQuant-4bBackbone=LLaMA-7b, Bits Per FPN=4.00-4.022024.05 | 78.62 | |
| CQ-2c8bBackbone=LLaMA-2-13b, Bits Per FPN=4.002024.05 | 78.62 | |
| CQ-2c8bBackbone=LLaMA-7b, Bits Per FPN=4.002024.05 | 78.61 | |
| CAR-DeBERTa-v3-LZero-shot=true, CSKB=ATOMIC2024.01 | 78.6 | |
| CAR-DeBERTa-v3-LZero-shot=true, CSKB=AbsATM2024.01 | 78.6 | |
| LLRCBackbone=Llama-2-13b, Param Ratio=0.90, Fine-tuning=None2025.12 | 78.6 | |
| KVQuant-2b-1%Backbone=LLaMA-13b, Bits Per FPN=2.32-2.352024.05 | 78.51 | |
| KVQuant-2b-1%Backbone=LLaMA-2-13b, Bits Per FPN=2.32-2.352024.05 | 78.51 | |
| KVQuant-4b-1%Backbone=LLaMA-7b, Bits Per FPN=4.32-4.352024.05 | 78.4 | |
| LLM-PrunerBackbone=Llama-2-13b, Param Ratio=0.80, Fine-tuning=None2025.12 | 78.3 | |
| CQ-4c8bBackbone=LLaMA-13b, Bits Per FPN=2.002024.05 | 78.29 | |
| BaselineBackbone=Llama-2-7b, Param Ratio=1.00, Fine-tuning=None2025.12 | 78.1 | |
| LLRCBackbone=Llama-2-7b, Param Ratio=0.90, Fine-tuning=None2025.12 | 78.1 | |
| LLM-Pruner+FinetuneBackbone=Llama-2-7b, Param Ratio=0.90, Fine-tuning=Alpaca2025.12 | 78.1 | |
| FP16Backbone=LLaMA-2-7b, Bits Per FPN=162024.05 | 78.07 | |
| KVQuant-4b-1%Backbone=LLaMA-2-7b, Bits Per FPN=4.32-4.352024.05 | 78.07 | |
| DeBERTa-v3-L (MR)Zero-shot=true, CSKB=ATOMIC2024.01 | 78 | |
| CQ-2c8bBackbone=LLaMA-2-7b, Bits Per FPN=4.002024.05 | 77.91 | |
| LLRCBackbone=Llama-2-13b, Param Ratio=0.80, Fine-tuning=None2025.12 | 77.9 | |
| KVQuant-4bBackbone=LLaMA-2-7b, Bits Per FPN=4.00-4.022024.05 | 77.86 | |
| LLM-Pruner+FinetuneBackbone=Llama-2-7b, Param Ratio=0.80, Fine-tuning=Alpaca2025.12 | 77.8 | |
| KVQuant-2b-1%Backbone=LLaMA-7b, Bits Per FPN=2.32-2.352024.05 | 77.69 | |
| VERA-T5-xxl (CANDLE Distilled)Zero-shot=true, CSKB=CANDLE2024.01 | 77.6 | |
| SVD-LLM+FinetuneBackbone=Llama-2-13b, Param Ratio=0.90, Fine-tuning=Alpaca2025.12 | 77.6 | |
| CQ-4c8bBackbone=LLaMA-2-13b, Bits Per FPN=2.002024.05 | 77.42 | |
| CQ-8c10bBackbone=Mistral-7b, Bits Per FPN=1.252024.05 | 77.31 | |
| LLM-PrunerBackbone=Llama-2-7b, Param Ratio=0.90, Fine-tuning=None2025.12 | 77.3 | |
| VERA-T5-xxlZero-shot=true, CSKB=AbsATM2024.01 | 77.2 | |
| KVQuant-2b-1%Backbone=LLaMA-2-7b, Bits Per FPN=2.32-2.352024.05 | 76.6 | |
| VERA-T5-xxlZero-shot=true, CSKB=ATOMIC2024.01 | 76.4 | |
| SVD-LLM+FinetuneBackbone=Llama-2-13b, Param Ratio=0.80, Fine-tuning=Alpaca2025.12 | 76.3 | |
| LLM-PrunerBackbone=Llama-2-7b, Param Ratio=0.80, Fine-tuning=None2025.12 | 76.2 | |
| CQ-4c8bBackbone=LLaMA-2-7b, Bits Per FPN=2.002024.05 | 76.12 | |
| CQ-4c8bBackbone=LLaMA-7b, Bits Per FPN=2.002024.05 | 76.11 | |
| CQ-8c10bBackbone=LLaMA-13b, Bits Per FPN=1.252024.05 | 75.9 | |
| KVQuant-2bBackbone=Mistral-7b, Bits Per FPN=2.00-2.022024.05 | 75.46 | |
| KVQuant-1b-1%Backbone=LLaMA-13b, Bits Per FPN=1.32-1.352024.05 | 75.46 | |
| SVD-LLM+FinetuneBackbone=Llama-2-7b, Param Ratio=0.90, Fine-tuning=Alpaca2025.12 | 75.3 | |
| CQ-8c8bBackbone=Mistral-7b, Bits Per FPN=1.002024.05 | 75.24 | |
| ChatGPTZero-shot=true, Variant=gpt-3.5-turbo2024.01 | 75.1 | |
| VERA-T5-xxlZero-shot=true, CSKB=ATM10X2024.01 | 75.1 | |
| LLRCBackbone=Llama-2-7b, Param Ratio=0.80, Fine-tuning=None2025.12 | 75.1 | |
| KVQuant-2bBackbone=LLaMA-13b, Bits Per FPN=2.00-2.022024.05 | 74.81 | |
| CQ-8c10bBackbone=LLaMA-2-13b, Bits Per FPN=1.252024.05 | 74.37 | |
| CQ-8c8bBackbone=LLaMA-13b, Bits Per FPN=1.002024.05 | 73.99 | |
| SVD-LLM (w)Backbone=Llama-2-13b, Param Ratio=0.90, Fine-tuning=None2025.12 | 73.9 | |
| KVQuant-1b-1%Backbone=Mistral-7b, Bits Per FPN=1.32-1.352024.05 | 73.83 | |
| CQ-8c10bBackbone=LLaMA-7b, Bits Per FPN=1.252024.05 | 73.45 | |
| CQ-8c10bBackbone=LLaMA-2-7b, Bits Per FPN=1.252024.05 | 73.07 | |
| CQ-8c8bBackbone=LLaMA-2-13b, Bits Per FPN=1.002024.05 | 73.01 | |
| GPT-4Zero-shot=true, Variant=gpt-42024.01 | 73 | |
| KVQuant-2bBackbone=LLaMA-7b, Bits Per FPN=2.00-2.022024.05 | 72.47 | |
| SVD-LLM+FinetuneBackbone=Llama-2-7b, Param Ratio=0.80, Fine-tuning=Alpaca2025.12 | 72.4 | |
| DeepSeekMoE# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.3B, FLOPs per 2K Tokens=4.3T, # Training Tokens=100B2024.01 | 72.3 | |
| KVQuant-1b-1%Backbone=LLaMA-7b, Bits Per FPN=1.32-1.352024.05 | 71.38 | |
| CQ-8c8bBackbone=LLaMA-2-7b, Bits Per FPN=1.002024.05 | 71.22 | |
| CQ-8c8bBackbone=LLaMA-7b, Bits Per FPN=1.002024.05 | 71.16 | |
| STL-AdapterZero-shot=true, CSKB=ATOMIC2024.01 | 71.1 | |
| KVQuant-1b-1%Backbone=LLaMA-2-13b, Bits Per FPN=1.32-1.352024.05 | 70.89 | |
| GShard# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.3B, FLOPs per 2K Tokens=4.3T, # Training Tokens=100B2024.01 | 70.6 | |
| Switch# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.01 | 70.5 | |
| AMOOptimizer=AMO, Model=Llama3.1-1.4B, Few-shot=02026.05 | 70.5 | |
| Mistral (Full-Attention)Model Scale=1.4B2024.07 | 70.4 | |
| Mamba (SSM)Model Scale=1.4B2024.07 | 70.3 | |
| Self-talkZero-shot=true2024.01 | 70.2 | |
| BMoJo (Fading)Model Scale=1.4B2024.07 | 70 | |
| KVQuant-1b-1%Backbone=LLaMA-2-7b, Bits Per FPN=1.32-1.352024.05 | 69.91 | |
| BMoJo (Fading + Eidetic)Model Scale=1.4B2024.07 | 69.9 | |
| SVD-LLM (w)Backbone=Llama-2-13b, Param Ratio=0.80, Fine-tuning=None2025.12 | 69.8 | |
| SVD-LLM (w)Backbone=Llama-2-7b, Param Ratio=0.90, Fine-tuning=None2025.12 | 69.4 | |
| SmolLM Ceiling (O(L^2))Parameters=135M, Training Tokens=∼600 Billion, Evaluation Mode=Zero-shot2026.04 | 68.55 | |
| Hash Layer# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.01 | 68.4 | |
| GPT 3.5Zero-shot=true, Variant=text-davinci-0032024.01 | 67.8 | |
| AMOOptimizer=AMO, Model=Llama3.1-760M, Few-shot=02026.05 | 67.7 | |
| ROBERTa-LZero-shot=true2024.01 | 67.6 |