Zero-shot Reasoning on PIQA
82.54PIQA Zero-shot AccuracyFP16
Evaluation results
| Method | Links | |
|---|---|---|
| FP16Model=Mistral, Precision=FP16, Sequence Length=20482025.07 | 82.54 | |
| FP16Backbone=Llama-3-70B, Bits=FP162026.06 | 82.37 | |
| LiftQuantBackbone=Llama-2-70B, Bits=2.412026.06 | 82.32 | |
| QTIPBackbone=Llama-2-70B, Bits=2.002026.06 | 82.16 | |
| FP16Backbone=Llama-2-70B, Bits=FP162026.06 | 82.15 | |
| OASISModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 82.1 | |
| OASIS-SModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 81.77 | |
| LiftQuantBackbone=Llama-3-70B, Bits=2.012026.06 | 81.72 | |
| QuIP#Backbone=Llama-2-70B, Bits=2.002026.06 | 81.39 | |
| LiftQuantBackbone=Llama-2-70B, Bits=2.012026.06 | 81.28 | |
| OPTQWBits=162023.07 | 80.9 | |
| QuIPWBits=162023.07 | 80.9 | |
| AtomModel=Mistral, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 80.71 | |
| FP16Model=LLaMA-3-8B, Precision=FP16, Sequence Length=20482025.07 | 80.63 | |
| OPTQWBits=42023.07 | 80.52 | |
| QuIPWBits=42023.07 | 80.47 | |
| VPTQBackbone=Llama-3-70B, Bits=2.022026.06 | 80.36 | |
| QuIPWBits=32023.07 | 80.25 | |
| QuaRotModel=Mistral, Precision=W4A4, Sequence Length=20482025.07 | 80.19 | |
| EQAT-g64Backbone=Llama-2-70B, Bits=2.262026.06 | 80.14 | |
| OASISModel=Mistral, Precision=W4A3, Sequence Length=20482025.07 | 79.87 | |
| FP16Backbone=Llama-3-8B, Bits=FP162026.06 | 79.6 | |
| FP16Backbone=Llama-2-13B, Bits=FP162026.06 | 79.05 | |
| BaselineRatio=1.0, Memory Footprint (GB)=12.6GB, Zero-shot=true2026.04 | 79 | |
| FP16Model=LLaMA-2-7B, Precision=FP16, Sequence Length=20482025.07 | 78.67 | |
| OASISModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 78.67 | |
| Llama2-7BBackbone=Llama2-7B2026.02 | 78.6 | |
| OPTQWBits=32023.07 | 78.56 | |
| FP16Backbone=Llama-2-7B, Bits=FP162026.06 | 78.07 | |
| DensePrune Rate=0%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 78.02 | |
| OASISModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 77.97 | |
| QuIP#Backbone=Llama-2-13B, Bits=2.002026.06 | 77.97 | |
| LiftQuantBackbone=Llama-2-13B, Bits=2.022026.06 | 77.91 | |
| QTIPBackbone=Llama-2-13B, Bits=2.002026.06 | 77.69 | |
| OASIS-SModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 77.62 | |
| LiftQuantBackbone=Llama-2-13B, Bits=2.422026.06 | 77.48 | |
| EQAT-g64Backbone=Llama-3-70B, Bits=2.252026.06 | 77.37 | |
| OASIS-SModel=Mistral, Precision=W4A3, Sequence Length=20482025.07 | 77.35 | |
| OASIS-SModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 77.31 | |
| OASISModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.07 | 77.09 | |
| EQAT-g64Backbone=Llama-2-13B, Bits=2.252026.06 | 77.04 | |
| LiftQuantBackbone=Llama-2-7B, Bits=2.422026.06 | 76.99 | |
| LiftQuantBackbone=Llama-2-7B, Bits=2.022026.06 | 76.55 | |
| LiftQuantBackbone=Llama-3-8B, Bits=2.022026.06 | 76.55 | |
| COMPOTBackbone=Llama2-7B, Training-free=true, CR=0.22026.02 | 76.4 | |
| QuaRotModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=20482025.07 | 76.39 | |
| QTIPBackbone=Llama-2-7B, Bits=2.002026.06 | 76.28 | |
| EQAT-g64Backbone=Llama-3-8B, Bits=2.252026.06 | 76.03 | |
| OASISModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.07 | 75.84 | |
| OASIS-SModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=20482025.07 | 75.82 | |
| PG pruningPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 75.41 | |
| EQAT-g64Backbone=Llama-2-7B, Bits=2.252026.06 | 75.3 | |
| AtomModel=LLaMA-2-7B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 75.14 | |
| OASIS-SModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=20482025.07 | 75.14 | |
| VPTQBackbone=Llama-3-8B, Bits=2.072026.06 | 75.12 | |
| QuIPWBits=22023.07 | 75.08 | |
| QuIP#Backbone=Llama-2-7B, Bits=2.002026.06 | 74.16 | |
| BonsaiPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 73.94 | |
| AtomModel=Mistral, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 73.69 | |
| Nirvana (Ours)Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 73.67 | |
| Nirvana-noTriggerSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 73.12 | |
| Swift-SVDRatio=0.8, Memory Footprint (GB)=10.1 GB, Zero-shot=true2026.04 | 73 | |
| Swift-SVD*Ratio=0.8, Memory Footprint (GB)=10.1 GB, Zero-shot=true2026.04 | 73 | |
| AtomModel=LLaMA-3-8B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 72.86 | |
| Gated DeltaNet-H1Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 72.57 | |
| SliceGPTPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 72.31 | |
| Gated DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 72.25 | |
| Gated DeltaNet-H2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 72.19 | |
| Dobi-SVD(w)Ratio=0.8, Memory Footprint (GB)=10.1 GB, Zero-shot=true2026.04 | 72 | |
| Mamba2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 71.87 | |
| LLM-PrunerPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 71.81 | |
| Wanda-spPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 71.6 | |
| MambaSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 71.32 | |
| PG pruningPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 71.11 | |
| AtomModel=LLaMA-2-7B, Precision=W4A3, Sequence Length=2048, Group Size=1282025.07 | 71.01 | |
| SVD-LLM(W)Ratio=0.8, Memory Footprint (GB)=10.1 GB, Zero-shot=true2026.04 | 71 | |
| SambaSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 70.94 | |
| DeltaNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 70.72 | |
| HGRN2Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 70.45 | |
| RetNetSetting=Zero-shot, Model Size=1.3B parameters2025.10 | 70.07 | |
| Transformer++Setting=Zero-shot, Model Size=1.3B parameters2025.10 | 70.02 | |
| AtomModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=2048, Group Size=1282025.07 | 69.45 | |
| BonsaiPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 68.44 | |
| QuaRotModel=LLaMA-3-8B, Precision=W4A4, Sequence Length=20482025.07 | 68.28 | |
| ASVDRatio=0.8, Memory Footprint (GB)=10.1 GB, Zero-shot=true2026.04 | 68 | |
| LLM-PrunerPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 67.52 | |
| COMPOTBackbone=Llama2-7B, Training-free=true, CR=0.42026.02 | 67 | |
| Swift-SVD*Ratio=0.6, Memory Footprint (GB)=7.7 GB, Zero-shot=true2026.04 | 67 | |
| Swift-SVDRatio=0.6, Memory Footprint (GB)=7.7 GB, Zero-shot=true2026.04 | 66 | |
| SliceGPTPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 65.4 | |
| Dobi-SVDBackbone=Llama2-7B, Training-free=false, CR=0.22026.02 | 65.2 | |
| Wanda-spPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 64.63 | |
| Dobi-SVD(w)Ratio=0.6, Memory Footprint (GB)=7.7 GB, Zero-shot=true2026.04 | 64 | |
| SVD-LLM(W)Ratio=0.6, Memory Footprint (GB)=7.7 GB, Zero-shot=true2026.04 | 63 | |
| PG pruningPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 61.8 | |
| LLM-PrunerPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 59.52 | |
| BonsaiPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 59.52 | |
| SliceGPTPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 59.47 | |
| Dobi-SVDBackbone=Llama2-7B, Training-free=false, CR=0.42026.02 | 57.3 | |
| Swift-SVD*Ratio=0.4, Memory Footprint (GB)=5.3 GB, Zero-shot=true2026.04 | 57 |