Language Understanding on ARC, COPA, BoolQ, PIQA, StoryCloze, RTE, MMLU Suite
49.57ARC (Challenge)Floating Point
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Floating PointModel=LLaMA-2-70B, Bits=16, Evaluation Protocol=Zero-shot2024.06 | 49.57 | 76.14 | 90 | 82.57 | 80.79 | 78.61 | 68.23 | — | 72.89 | |
| ShiftAddLLMModel=LLaMA-2-70B, Bits=3, Evaluation Protocol=Zero-shot2024.06 | 48.38 | 77.06 | 93 | 84.25 | 80.47 | 78.49 | 75.09 | — | 74.88 | |
| LUT-GEMMModel=LLaMA-2-70B, Bits=3, Evaluation Protocol=Zero-shot2024.06 | 47.7 | 76.42 | 89 | 80.31 | 80.2 | 77.78 | 68.59 | — | — | |
| OPTQModel=LLaMA-2-70B, Bits=3, Evaluation Protocol=Zero-shot2024.06 | 45.82 | 76.34 | 90 | 81.74 | 79.71 | 77.34 | 67.51 | — | 72.33 | |
| Floating PointModel=OPT-66B, Bits=16, Evaluation Protocol=Zero-shot2024.06 | 37.2 | 71.25 | 86 | 69.82 | 78.67 | 77.47 | 60.65 | — | 63.37 | |
| ShiftAddLLMModel=OPT-66B, Bits=3, Evaluation Protocol=Zero-shot2024.06 | 35.24 | 70.88 | 87 | 72.45 | 77.64 | 77.15 | 63.18 | — | 63.89 | |
| OPTQModel=OPT-66B, Bits=3, Evaluation Protocol=Zero-shot2024.06 | 24.66 | 48.86 | 70 | 52.05 | 64.47 | 67.09 | 53.07 | — | 50.52 | |
| LUT-GEMMModel=OPT-66B, Bits=3, Evaluation Protocol=Zero-shot2024.06 | 24.15 | 51.85 | 81 | 53.52 | 61.97 | 60.6 | 48.74 | — | 50.7 |