Question Answering and Commonsense Reasoning on LM Eval ARCC, ARCE, HellaSwag, PIQA (test)
61.6ARCCMeta 'FP8'
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Meta 'FP8'Bits=16, Model=Llama 3.1 405B Instruct, Evaluation Protocol=Zero-shot2024.06 | 61.6 | 81.4 | 67.1 | 83.8 | |
| QTIPBits=3, Model=Llama 3.1 405B Instruct, Evaluation Protocol=Zero-shot2024.06 | 61.5 | 81.4 | 66.8 | 83.5 | |
| QTIPBits=4, Model=Llama 3.1 405B Instruct, Evaluation Protocol=Zero-shot2024.06 | 61.3 | 80.9 | 66.7 | 84.2 | |
| QTIPBits=2, Model=Llama 3.1 405B Instruct, Evaluation Protocol=Zero-shot2024.06 | 60.7 | 81.1 | 65.4 | 82.2 | |
| BF16Bits=16, Model=Llama 3.1 70B Instruct, Evaluation Protocol=Zero-shot2024.06 | 56.7 | 75.6 | 61.5 | 82.8 | |
| QTIPBits=4, Model=Llama 3.1 70B Instruct, Evaluation Protocol=Zero-shot2024.06 | 56.3 | 75.8 | 61.4 | 83 | |
| QTIPBits=3, Model=Llama 3.1 70B Instruct, Evaluation Protocol=Zero-shot2024.06 | 55.1 | 75.1 | 60.8 | 82.6 | |
| QTIPBits=2, Model=Llama 3.1 70B Instruct, Evaluation Protocol=Zero-shot2024.06 | 54.4 | 72.6 | 59.4 | 82.5 | |
| PV-TuningBits=2.01, Model=Llama 3.1 70B Instruct, Evaluation Protocol=Zero-shot2024.06 | 52.7 | 72.2 | 60.2 | 82.6 | |
| BF16Bits=16, Model=Llama 3.1 8B Instruct, Evaluation Protocol=Zero-shot2024.06 | 51.6 | 77.8 | 57.7 | 80 | |
| QTIPBits=4, Model=Llama 3.1 8B Instruct, Evaluation Protocol=Zero-shot2024.06 | 50.7 | 78 | 57.5 | 80.1 | |
| QTIPBits=3, Model=Llama 3.1 8B Instruct, Evaluation Protocol=Zero-shot2024.06 | 50.4 | 77.7 | 56.9 | 79.3 | |
| PV-TuningBits=2.07, Model=Llama 3.1 8B Instruct, Evaluation Protocol=Zero-shot2024.06 | 46.2 | 75.4 | 54.4 | 78.7 | |
| QTIPBits=2, Model=Llama 3.1 8B Instruct, Evaluation Protocol=Zero-shot2024.06 | 45.1 | 75.6 | 54.5 | 79 | |
| QTIPBackbone Model=Llama 3.2 3B, Size (GB)=2.1, Precision=4-bit2024.06 | 43.5 | 74.3 | 51.9 | 75.1 | |
| BF16Backbone Model=Llama 3.2 3B, Size (GB)=6, Precision=16-bit2024.06 | 43.3 | 74.3 | 52.2 | 75.7 | |
| BF16Backbone Model=Llama 3.2 1B, Size (GB)=2.4, Precision=16-bit2024.06 | 36 | 68.5 | 45.2 | 74.2 | |
| QTIPBackbone Model=Llama 3.2 1B, Size (GB)=0.97, Precision=4-bit2024.06 | 34.8 | 68.4 | 44.5 | 73.3 |