Zero-shot Common Sense Reasoning on CSR Suite (ARC-E, ARC-C, BoolQ, PIQA, SIQA, HellaSwag, OBQA, WinoGrande)
68.95AccuracyRTN
Evaluation Results
| Method | Links | |
|---|---|---|
| RTNModel=LLaMA-2-70B, Precision=FP162026.06 | 68.95 | |
| QuaRot-RTNModel=LLaMA-2-70B, Precision=FP162026.06 | 68.91 | |
| InsertQuantModel=LLaMA-2-70B, Precision=FP162026.06 | 68.84 | |
| PrefixQuantModel=LLaMA-2-70B, Precision=FP162026.06 | 68.39 | |
| QuaRot-RTNModel=Mistral-7B-v0.3, Precision=FP162026.06 | 66.92 | |
| RTNModel=Mistral-7B-v0.3, Precision=FP162026.06 | 66.89 | |
| PrefixQuantModel=Mistral-7B-v0.3, Precision=FP162026.06 | 66.33 | |
| InsertQuantModel=Mistral-7B-v0.3, Precision=FP162026.06 | 65.84 | |
| PrefixQuantModel=LLaMA-2-70B, Precision=W4A4sKV42026.06 | 65.56 | |
| InsertQuantModel=LLaMA-2-70B, Precision=W4A4sKV42026.06 | 65.22 | |
| PrefixQuantModel=Mistral-7B-v0.3, Precision=W4A4sKV42026.06 | 64.26 | |
| InsertQuantModel=Mistral-7B-v0.3, Precision=W4A4sKV42026.06 | 62.75 | |
| PrefixQuantModel=LLaMA-2-7B, Precision=FP162026.06 | 62.5 | |
| RTNModel=LLaMA-2-7B, Precision=FP162026.06 | 62.44 | |
| QuaRot-RTNModel=LLaMA-2-7B, Precision=FP162026.06 | 62.44 | |
| InsertQuantModel=LLaMA-2-7B, Precision=FP162026.06 | 62.44 | |
| QuaRot-RTNModel=LLaMA-3.2-3B, Precision=FP162026.06 | 61.04 | |
| PrefixQuantModel=LLaMA-3.2-3B, Precision=FP162026.06 | 61.01 | |
| RTNModel=LLaMA-3.2-3B, Precision=FP162026.06 | 60.99 | |
| InsertQuantModel=LLaMA-3.2-3B, Precision=FP162026.06 | 60.91 | |
| InsertQuantModel=LLaMA-2-7B, Precision=W4A4sKV42026.06 | 60.26 | |
| PrefixQuantModel=LLaMA-2-7B, Precision=W4A4sKV42026.06 | 60.17 | |
| InsertQuantModel=LLaMA-3.2-3B, Precision=W4A4sKV42026.06 | 57.19 | |
| PrefixQuantModel=LLaMA-3.2-3B, Precision=W4A4sKV42026.06 | 57.06 | |
| QuaRot-RTNModel=LLaMA-2-70B, Precision=W4A4sKV42026.06 | 53.49 | |
| QuaRot-RTNModel=Mistral-7B-v0.3, Precision=W4A4sKV42026.06 | 40.18 | |
| QuaRot-RTNModel=LLaMA-3.2-3B, Precision=W4A4sKV42026.06 | 38.42 | |
| RTNModel=LLaMA-2-70B, Precision=W4A4sKV42026.06 | 36.22 | |
| RTNModel=LLaMA-2-7B, Precision=W4A4sKV42026.06 | 35.56 | |
| RTNModel=Mistral-7B-v0.3, Precision=W4A4sKV42026.06 | 34.92 | |
| RTNModel=LLaMA-3.2-3B, Precision=W4A4sKV42026.06 | 34.57 | |
| QuaRot-RTNModel=LLaMA-2-7B, Precision=W4A4sKV42026.06 | 34.45 |