Language Understanding on MMLU (Category Breakdown Scores)
37.31STEM Accuracy (MMLU)FP16
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| FP16Backbone=LLaMA2-7B, Avg. Bit=16, Evaluation Protocol=five-shot2024.10 | 37.31 | 52.91 | 43.04 | 53.82 | 46.46 | |
| CMPQBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=five-shot2024.10 | 36.48 | 50.57 | 41.91 | 52.16 | 45 | |
| QuIPBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=five-shot2024.10 | 36.25 | 48.07 | 39.83 | 49.07 | 43 | |
| AWQBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=five-shot2024.10 | 36.24 | 50.31 | 41.72 | 52.22 | 44.85 | |
| RTNBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=five-shot2024.10 | 34.59 | 46.05 | 39.62 | 47.78 | 41.83 | |
| GPTQBackbone=LLaMA2-7B, Avg. Bit=4, Evaluation Protocol=five-shot2024.10 | 34.13 | 43.87 | 38.13 | 45.59 | 40.25 | |
| CMPQBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=five-shot2024.10 | 32.54 | 43.26 | 36.34 | 45.99 | 39.27 | |
| AWQBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=five-shot2024.10 | 31.74 | 36.59 | 30.86 | 38.09 | 33.98 | |
| GPTQBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=five-shot2024.10 | 28.24 | 25.67 | 31.69 | 32.92 | 29.85 | |
| RTNBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=five-shot2024.10 | 27.93 | 23.63 | 24.87 | 24.12 | 25.08 | |
| QuIPBackbone=LLaMA2-7B, Avg. Bit=3, Evaluation Protocol=five-shot2024.10 | 27.27 | 30.81 | 27.08 | 25.51 | 27.57 |