Commonsense Reasoning on Common Sense QA (test) (5-shot Aggregate)
58ARC-C Accuracy (5-shot)QA-LoRA
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| QA-LoRAFoundation Model=LLaMA2-13B, Bit Constraint=4, Evaluation Protocol=5-shot2024.05 | 58 | 79.2 | 81.3 | 74 | 83.3 | 83.8 | 49.4 | 75.2 | 1.43 | |
| LLaMA2-13BFoundation Model=LLaMA2-13B, Bit Constraint=16, Evaluation Protocol=5-shot2024.05 | 57.5 | 81.7 | 81.7 | 76 | 84.4 | 83.2 | 48.2 | 75.9 | 1.6 | |
| GPTQFoundation Model=LLaMA2-13B, Bit Constraint=4, Evaluation Protocol=5-shot2024.05 | 56.5 | 81.1 | 80.9 | 75.6 | 83.3 | 81.7 | 47.4 | 75 | 1.58 | |
| LLM-QFAFoundation Model=LLaMA2-13B, Bit Constraint=4, Evaluation Protocol=5-shot2024.05 | 56 | 79.6 | 82 | 73.2 | 83.5 | 83.2 | 51 | 75.4 | 1.31 | |
| QA-LoRAFoundation Model=LLaMA2-7B, Bit Constraint=4, Evaluation Protocol=5-shot2024.05 | 55.5 | 79 | 80 | 73.3 | 79.6 | 75.9 | 46.4 | 72.4 | 1.4 | |
| LLM-QFAFoundation Model=LLaMA2-7B, Bit Constraint=4, Evaluation Protocol=5-shot2024.05 | 53.8 | 76.8 | 79.3 | 73.5 | 78.1 | 77.4 | 49 | 72.4 | 1.12 | |
| LLM-QFAFoundation Model=LLaMA2-13B, Bit Constraint=3, Evaluation Protocol=5-shot2024.05 | 53.7 | 75.1 | 79.7 | 70.3 | 80.5 | 78.4 | 48 | 72 | 1.27 | |
| QA-LoRAFoundation Model=LLaMA2-13B, Bit Constraint=3, Evaluation Protocol=5-shot2024.05 | 53.5 | 67 | 79.4 | 66.7 | 80.1 | 76.3 | 41.8 | 68.5 | 1.72 | |
| LLaMA2-7BFoundation Model=LLaMA2-7B, Bit Constraint=16, Evaluation Protocol=5-shot2024.05 | 52 | 78.2 | 80.1 | 74.1 | 81.1 | 79.3 | 45.2 | 73 | 1.59 | |
| GPTQFoundation Model=LLaMA2-7B, Bit Constraint=4, Evaluation Protocol=5-shot2024.05 | 50.8 | 77 | 79.5 | 73.8 | 80.2 | 74.1 | 43.4 | 71.3 | 1.61 | |
| LLM-QFAFoundation Model=LLaMA2-13B, Bit Constraint=2, Evaluation Protocol=5-shot2024.05 | 49.2 | 70.9 | 77 | 67.2 | 76.3 | 74.3 | 44.6 | 68.4 | 1.24 | |
| LLM-QFAFoundation Model=LLaMA2-7B, Bit Constraint=3, Evaluation Protocol=5-shot2024.05 | 49.1 | 72.3 | 76.7 | 69 | 73.8 | 72.8 | 43.4 | 68 | 1.26 | |
| QA-LoRAFoundation Model=LLaMA2-13B, Bit Constraint=2, Evaluation Protocol=5-shot2024.05 | 49.1 | 70.8 | 76.6 | 66.4 | 76.1 | 74.1 | 44.8 | 68.1 | 1.21 | |
| QA-LoRAFoundation Model=LLaMA2-7B, Bit Constraint=3, Evaluation Protocol=5-shot2024.05 | 47.8 | 72.4 | 75 | 68.4 | 73.6 | 72 | 44.8 | 67.7 | 1.08 | |
| GPTQFoundation Model=LLaMA2-13B, Bit Constraint=3, Evaluation Protocol=5-shot2024.05 | 47.8 | 68.6 | 77.7 | 67.9 | 77.1 | 71.9 | 42.8 | 67.7 | 1.38 | |
| LLM-QFAFoundation Model=LLaMA2-7B, Bit Constraint=2, Evaluation Protocol=5-shot2024.05 | 43.1 | 64.8 | 73.2 | 62.2 | 67 | 64.3 | 38.8 | 61.7 | 1.16 | |
| QA-LoRAFoundation Model=LLaMA2-7B, Bit Constraint=2, Evaluation Protocol=5-shot2024.05 | 40.4 | 65.6 | 73.6 | 62 | 66 | 65.9 | 37.2 | 61.7 | 1.32 | |
| GPTQFoundation Model=LLaMA2-7B, Bit Constraint=3, Evaluation Protocol=5-shot2024.05 | 30.1 | 49.9 | 68.3 | 59.3 | 55.5 | 44.3 | 35 | 52.1 | 1.13 | |
| GPTQFoundation Model=LLaMA2-13B, Bit Constraint=2, Evaluation Protocol=5-shot2024.05 | 27.8 | 25.8 | 50.2 | 50.2 | 26.6 | 37.8 | 23.4 | 35.7 | 1.26 | |
| GPTQFoundation Model=LLaMA2-7B, Bit Constraint=2, Evaluation Protocol=5-shot2024.05 | 25.8 | 26.2 | 51.1 | 50.6 | 26 | 41.7 | 25 | 36.8 | 1.31 |