Zero-shot QA and Reasoning on Accuracy Tasks (AC, AE, WI, QA)
62AC ScoreFP16
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| FP16Model=Mixtral-8x7B, Bit=162026.01 | 62 | 87.3 | 75.3 | 83.5 | |
| LoPRoModel=Mixtral-8x7B, Bit=3.172026.01 | 61 | 85.3 | 77 | 83.3 | |
| LoPRovModel=Mixtral-8x7B, Bit=3.172026.01 | 60.6 | 86.5 | 76.7 | 82.9 | |
| LoPRovModel=Mixtral-8x7B, Bit=2.172026.01 | 55.8 | 82.9 | 74 | 80.5 | |
| GPTVQModel=Mixtral-8x7B, Bit=3.132026.01 | 54.9 | 72.9 | 74.8 | 82.6 | |
| LoPRoModel=Mixtral-8x7B, Bit=2.172026.01 | 53.2 | 81.8 | 72 | 78.1 | |
| GPTQModel=Mixtral-8x7B, Bit=3.132026.01 | 52.1 | 69.3 | 74.4 | 80.9 | |
| FP16Model=LLaMA3-8B, Bit=162026.01 | 50.2 | 80.1 | 73.5 | 79.7 | |
| FP16Model=LLaMA2-13B, Bit=162026.01 | 49.1 | 77.4 | 73.9 | 81.4 | |
| LoPRovModel=LLaMA3-8B, Bit=3.172026.01 | 45.3 | 76.8 | 71.9 | 78.4 | |
| Quip#Model=LLaMA3-8B, Bit=3.002026.01 | 45.2 | 75.2 | 72.3 | 78 | |
| GPTVQModel=LLaMA3-8B, Bit=3.132026.01 | 45.1 | 76.7 | 72.5 | 77.8 | |
| LoPRoModel=LLaMA2-13B, Bit=3.172026.01 | 44.9 | 78.7 | 71.1 | 78.5 | |
| GPTVQModel=LLaMA2-13B, Bit=3.132026.01 | 44.5 | 75.2 | 72 | 77.8 | |
| LoPRovModel=LLaMA2-13B, Bit=3.172026.01 | 44.5 | 77.4 | 71 | 78.3 | |
| LoPRoModel=LLaMA3-8B, Bit=3.172026.01 | 44.1 | 75.9 | 73 | 77.5 | |
| FP16Model=LLaMA2-7B, Bit=162026.01 | 43.4 | 76.3 | 69.1 | 78.4 | |
| LQERModel=LLaMA2-13B, Bit=3.242026.01 | 42.3 | 76.7 | 71.2 | 77.4 | |
| Quip#Model=LLaMA2-13B, Bit=3.002026.01 | 42.2 | 76.6 | 71.5 | 77.6 | |
| OminiQModel=LLaMA3-8B, Bit=3.132026.01 | 42.2 | 72.4 | 71.3 | 75.5 | |
| OminiQModel=LLaMA2-13B, Bit=3.132026.01 | 42 | 77.9 | 71.3 | 78 | |
| GPTVQModel=Mixtral-8x7B, Bit=2.132026.01 | 42 | 71.6 | 66.5 | 75.9 | |
| LoPRoModel=LLaMA2-7B, Bit=3.172026.01 | 41 | 74.9 | 68.9 | 76.3 | |
| LoPRovModel=LLaMA2-7B, Bit=3.172026.01 | 41 | 74.8 | 69.1 | 76.7 | |
| OminiQModel=LLaMA2-7B, Bit=3.132026.01 | 40.8 | 74.5 | 67.5 | 77.7 | |
| LQERModel=LLaMA2-7B, Bit=3.282026.01 | 40.4 | 74.2 | 68.4 | 74.9 | |
| Quip#Model=LLaMA2-7B, Bit=3.002026.01 | 40.2 | 75.1 | 67 | 76.2 | |
| GPTVQModel=LLaMA2-7B, Bit=3.132026.01 | 39.9 | 74.1 | 69.1 | 76.2 | |
| LoPRovModel=LLaMA2-13B, Bit=2.172026.01 | 38.8 | 74.2 | 68 | 75.4 | |
| GPTVQModel=LLaMA2-13B, Bit=2.132026.01 | 38.7 | 73.6 | 68.5 | 75.4 | |
| GPTQModel=LLaMA2-13B, Bit=3.132026.01 | 38.5 | 65.7 | 63.9 | 76.5 | |
| GPTQModel=LLaMA3-8B, Bit=3.132026.01 | 37.7 | 70.5 | 71.1 | 74.9 | |
| LoPRovModel=LLaMA3-8B, Bit=2.172026.01 | 37 | 69.2 | 68 | 71.4 | |
| Quip#Model=LLaMA2-13B, Bit=2.002026.01 | 36.2 | 68.6 | 63.6 | 74.2 | |
| LoPRovModel=LLaMA2-7B, Bit=2.172026.01 | 34.6 | 69 | 66.5 | 72.7 | |
| LoPRoModel=LLaMA2-13B, Bit=2.172026.01 | 33.6 | 69 | 66.3 | 72.4 | |
| LQERModel=LLaMA2-13B, Bit=2.642026.01 | 33.2 | 65.8 | 66.4 | 73.1 | |
| LQERModel=LLaMA2-7B, Bit=2.802026.01 | 33.1 | 60.4 | 61.2 | 70.7 | |
| OminiQModel=LLaMA2-13B, Bit=2.132026.01 | 31.3 | 62.3 | 65.6 | 72.3 | |
| GPTVQModel=LLaMA3-8B, Bit=2.132026.01 | 31.3 | 57.3 | 68.3 | 67.8 | |
| LoPRoModel=LLaMA2-7B, Bit=2.172026.01 | 31.2 | 62.8 | 63.8 | 71.1 | |
| GPTQModel=LLaMA2-7B, Bit=3.132026.01 | 31.1 | 58.5 | 59.2 | 71.5 | |
| Quip#Model=LLaMA3-8B, Bit=2.002026.01 | 30.8 | 57.1 | 67 | 67.5 | |
| LoPRoModel=LLaMA3-8B, Bit=2.172026.01 | 30.5 | 58.9 | 62.2 | 68.2 | |
| GPTVQModel=LLaMA2-7B, Bit=2.132026.01 | 30.2 | 64.3 | 64.1 | 72.1 | |
| Quip#Model=LLaMA2-7B, Bit=2.002026.01 | 29.9 | 61.3 | 61.7 | 69.6 | |
| OminiQModel=LLaMA2-7B, Bit=2.132026.01 | 28.8 | 58.1 | 59.1 | 70.2 | |
| GPTQModel=Mixtral-8x7B, Bit=2.132026.01 | 26.6 | 35.7 | 49.5 | 57.7 | |
| GPTQModel=LLaMA2-13B, Bit=2.132026.01 | 23.3 | 43.3 | 54.7 | 61.3 | |
| GPTQModel=LLaMA3-8B, Bit=2.132026.01 | 21.1 | 29.3 | 52.1 | 54.4 | |
| GPTQModel=LLaMA2-7B, Bit=2.132026.01 | 20.9 | 34.9 | 52.3 | 57.2 | |
| OminiQModel=LLaMA3-8B, Bit=2.132026.01 | 19.3 | 36.1 | 51.9 | 59 |