Zero-shot Reasoning on (BoolQ, PIQA, HellaSwag, WinoGrande, ARC)
71.1BoolQ Accuracy (Zero-shot)FP16
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| FP16Model=LLaMA2-7B, Wbits=162025.05 | 71.1 | 76.88 | 72.94 | 67.09 | 53.58 | 40.61 | 63.7 | |
| FP16Model=LLaMA2-13B, Wbits=162025.05 | 68.99 | 79.05 | 76.62 | 69.77 | 57.95 | 44.2 | 66.1 | |
| MBOKModel=LLaMA2-7B, Wbits=2×12025.05 | 66.94 | 74.97 | 65.59 | 61.72 | 44.82 | 34.21 | 58.04 | |
| MBOKModel=LLaMA2-13B, Wbits=3×12025.05 | 66.8 | 77.59 | 73.79 | 65.27 | 54.47 | 41.89 | 63.3 | |
| MBOKModel=LLaMA2-13B, Wbits=2×12025.05 | 66.32 | 75.84 | 70.24 | 62.51 | 50 | 37.46 | 60.4 | |
| MoSModel=LLaMA2-13B, Wbits=12025.05 | 66.12 | 73.72 | 63.8 | 58.98 | 45.71 | 33.19 | 57.09 | |
| OneBitModel=LLaMA2-13B, Wbits=12025.05 | 65.66 | 71.6 | 60.07 | 56.91 | 45.76 | 31.74 | 55.29 | |
| MBOKModel=LLaMA2-7B, Wbits=3×12025.05 | 65.46 | 75.79 | 69.59 | 62.04 | 49.11 | 37.8 | 59.97 | |
| MoSModel=LLaMA2-7B, Wbits=12025.05 | 65.02 | 71.55 | 59.41 | 56.18 | 41.84 | 30.03 | 54.01 | |
| OneBitModel=LLaMA2-7B, Wbits=12025.05 | 63.06 | 70.4 | 54.24 | 56.67 | 40.82 | 29.35 | 52.42 | |
| BiLLMModel=LLaMA2-13B, Wbits=1.112025.05 | 62.2 | 62.51 | 38.05 | 56.35 | 40.69 | 27.73 | 47.92 | |
| PB-LLMModel=LLaMA2-7B, Wbits=1.72025.05 | 62.17 | 52.82 | 26.87 | 50.11 | 26.89 | 24.31 | 40.53 | |
| BiLLMModel=LLaMA2-7B, Wbits=1.112025.05 | 62.14 | 59.19 | 35.18 | 53.11 | 34.22 | 26.54 | 45.06 | |
| OmniQuantModel=LLaMA2-13B, Wbits=22025.05 | 62.05 | 62.24 | 50.34 | 53.2 | 40.66 | 29.61 | 49.68 | |
| LLM-QATModel=LLaMA2-7B, Wbits=22025.05 | 59.14 | 50.12 | 25.1 | 49.08 | 26.26 | 26.96 | 35.89 | |
| OmniQuantModel=LLaMA2-7B, Wbits=22025.05 | 58.69 | 56.53 | 33.87 | 51.22 | 33.63 | 24.32 | 43.12 | |
| OPTQModel=LLaMA2-7B, Wbits=22025.05 | 42.97 | 49.46 | 26.19 | 50.28 | 26.77 | 28.58 | 37.38 | |
| OPTQModel=LLaMA2-13B, Wbits=22025.05 | 40.61 | 51.74 | 25.67 | 51.85 | 25.46 | 27.3 | 37.11 | |
| LLM-QATModel=LLaMA2-13B, Wbits=22025.05 | 39.85 | 49.08 | 24.37 | 51.38 | 27.15 | 24.32 | 36.03 | |
| PB-LLMModel=LLaMA2-13B, Wbits=1.72025.05 | 37.82 | 53.26 | 28.89 | 49.48 | 28.28 | 23.72 | 36.91 |