Question Answering on ARC, HellaSwag, and MMLU (Zero-shot, test)
52.9ARC-CFP16
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| FP16Backbone Model=LLaMA-30B, Quantization Precision=FP162024.02 | 52.9 | 78.96 | 82.63 | 54.66 | 67.29 | |
| FP16Model=LLaMA-30B, Weight Precision=FP162024.02 | 52.9 | 78.96 | 82.63 | 54.66 | 67.29 | |
| Z-FOLDBackbone Model=LLaMA-30B, Quantization Precision=INT32024.02 | 50.34 | 75.84 | 79.69 | 51 | 64.22 | |
| aespaBackbone Model=LLaMA-30B, Quantization Precision=INT32024.02 | 50.34 | 77.53 | 79.79 | 50.55 | 64.55 | |
| AffineQuantBackbone Model=LLaMA-30B, Quantization Precision=INT32024.02 | 49.66 | 77.1 | 79.49 | 50.37 | 64.16 | |
| OmniQuantBackbone Model=LLaMA-30B, Quantization Precision=INT32024.02 | 49.49 | 76.52 | 79.76 | 50.68 | 64.11 | |
| FP16Backbone Model=LLaMA2-13B, Quantization Precision=FP162024.02 | 49.06 | 77.44 | 79.39 | 52.1 | 64.5 | |
| FP16Model=LLaMA2-13B, Weight Precision=FP162024.02 | 49.06 | 77.44 | 79.39 | 52.1 | 64.5 | |
| FP16Backbone Model=LLaMA-13B, Quantization Precision=FP162024.02 | 47.87 | 74.75 | 79.08 | 43.46 | 61.29 | |
| FP16Model=LLaMA-13B, Weight Precision=FP162024.02 | 47.87 | 74.75 | 79.08 | 43.46 | 61.29 | |
| AffineQuantBackbone Model=LLaMA2-13B, Quantization Precision=INT32024.02 | 47.18 | 75.42 | 75.28 | 45.61 | 60.87 | |
| aespaBackbone Model=LLaMA2-13B, Quantization Precision=INT32024.02 | 46.84 | 75.25 | 75.78 | 47.09 | 61.24 | |
| Z-FOLDBackbone Model=LLaMA2-13B, Quantization Precision=INT32024.02 | 46.42 | 72.77 | 74.79 | 47.91 | 60.47 | |
| FP16Backbone Model=LLaMA2-7B, Quantization Precision=FP162024.02 | 46.16 | 74.49 | 75.99 | 41.87 | 59.63 | |
| FP16Model=LLaMA2-7B, Weight Precision=FP162024.02 | 46.16 | 74.49 | 75.99 | 41.87 | 59.63 | |
| aespaBackbone Model=LLaMA-13B, Quantization Precision=INT32024.02 | 45.82 | 71.8 | 75.87 | 38.63 | 58.03 | |
| OmniQuantBackbone Model=LLaMA2-13B, Quantization Precision=INT32024.02 | 45.65 | 74.33 | 74.77 | 43.92 | 59.67 | |
| OPTQBackbone Model=LLaMA-30B, Quantization Precision=INT32024.02 | 45.31 | 71.55 | 77.17 | 42.01 | 59.01 | |
| Z-FOLDBackbone Model=LLaMA-13B, Quantization Precision=INT32024.02 | 44.88 | 71 | 75.66 | 36.88 | 57.11 | |
| FP16Backbone Model=LLaMA-7B, Quantization Precision=FP162024.02 | 44.62 | 72.85 | 76.18 | 32.19 | 56.46 | |
| FP16Model=LLaMA-7B, Weight Precision=FP162024.02 | 44.62 | 72.85 | 76.18 | 32.19 | 56.46 | |
| OmniQuantBackbone Model=LLaMA-13B, Quantization Precision=INT32024.02 | 44.03 | 69.7 | 75.15 | 35.89 | 56.19 | |
| AffineQuantBackbone Model=LLaMA-13B, Quantization Precision=INT32024.02 | 43.6 | 70.24 | 75.1 | 32.67 | 55.4 | |
| OPTQBackbone Model=LLaMA2-13B, Quantization Precision=INT32024.02 | 43.09 | 70.45 | 72.02 | 39.37 | 56.23 | |
| OPTQBackbone Model=LLaMA-13B, Quantization Precision=INT32024.02 | 43 | 67.89 | 72.45 | 28.62 | 52.99 | |
| aespaBackbone Model=LLaMA2-7B, Quantization Precision=INT32024.02 | 41.38 | 69.11 | 71.78 | 38.18 | 55.11 | |
| Z-FOLDBackbone Model=LLaMA-7B, Quantization Precision=INT32024.02 | 41.21 | 66.92 | 72.5 | 28.9 | 52.38 | |
| aespaModel=LLaMA-30B, Weight Precision=INT22024.02 | 41.13 | 67 | 67.9 | 35.67 | 52.93 | |
| aespaBackbone Model=LLaMA-7B, Quantization Precision=INT32024.02 | 40.87 | 69.15 | 71.54 | 30.57 | 53.03 | |
| AffineQuantBackbone Model=LLaMA2-7B, Quantization Precision=INT32024.02 | 40.78 | 67.21 | 70.75 | 30.93 | 52.42 | |
| OmniQuantBackbone Model=LLaMA2-7B, Quantization Precision=INT32024.02 | 40.36 | 67.3 | 71 | 31.26 | 52.48 | |
| Z-FOLDBackbone Model=LLaMA2-7B, Quantization Precision=INT32024.02 | 40.1 | 64.65 | 69.92 | 33.69 | 52.09 | |
| Z-FOLDModel=LLaMA-30B, Weight Precision=INT22024.02 | 39.93 | 65.07 | 65.89 | 30.85 | 50.44 | |
| AffineQuantBackbone Model=LLaMA-7B, Quantization Precision=INT32024.02 | 39.25 | 65.61 | 70.56 | 29.68 | 51.28 | |
| OmniQuantBackbone Model=LLaMA-7B, Quantization Precision=INT32024.02 | 38.99 | 67.3 | 70.31 | 29.33 | 51.48 | |
| OPTQBackbone Model=LLaMA2-7B, Quantization Precision=INT32024.02 | 37.46 | 63.01 | 64.85 | 28.79 | 48.53 | |
| OPTQBackbone Model=LLaMA-7B, Quantization Precision=INT32024.02 | 36.95 | 62.63 | 68.33 | 25.51 | 48.36 | |
| RTNBackbone Model=LLaMA-13B, Quantization Precision=INT32024.02 | 36.09 | 56.23 | 62.03 | 26.2 | 45.14 | |
| aespaModel=LLaMA-13B, Weight Precision=INT22024.02 | 34.73 | 61.49 | 62.68 | 28.74 | 46.91 | |
| RTNBackbone Model=LLaMA2-13B, Quantization Precision=INT32024.02 | 34.56 | 55.98 | 59.44 | 25.45 | 43.86 | |
| OmniQuantModel=LLaMA-30B, Weight Precision=INT22024.02 | 34.22 | 58.5 | 64.83 | 25.91 | 45.87 | |
| aespaModel=LLaMA-7B, Weight Precision=INT22024.02 | 33.36 | 55.64 | 58.31 | 23.12 | 42.61 | |
| Z-FOLDModel=LLaMA-13B, Weight Precision=INT22024.02 | 32.68 | 58.08 | 57.89 | 26.44 | 43.77 | |
| AffineQuantModel=LLaMA-13B, Weight Precision=INT22024.02 | 32.17 | 56.36 | 60.29 | 25.22 | 43.51 | |
| aespaModel=LLaMA2-13B, Weight Precision=INT22024.02 | 31.91 | 55.18 | 55.49 | 29.97 | 43.14 | |
| AffineQuantModel=LLaMA2-13B, Weight Precision=INT22024.02 | 30.8 | 52.9 | 57.74 | 24.45 | 41.47 | |
| Z-FOLDModel=LLaMA-7B, Weight Precision=INT22024.02 | 30.63 | 52.44 | 53.55 | 23.27 | 39.97 | |
| aespaModel=LLaMA2-7B, Weight Precision=INT22024.02 | 30.29 | 51.47 | 56.75 | 25.59 | 41.03 | |
| OPTQModel=LLaMA-7B, Weight Precision=INT22024.02 | 29.18 | 26.14 | 26.18 | 24.04 | 26.39 | |
| RTNModel=LLaMA-7B, Weight Precision=INT22024.02 | 28.67 | 25 | 26.43 | 25.72 | 26.46 | |
| Z-FOLDModel=LLaMA2-13B, Weight Precision=INT22024.02 | 28.41 | 48.32 | 51.59 | 23.98 | 38.08 | |
| RTNModel=LLaMA2-7B, Weight Precision=INT22024.02 | 28.33 | 26.01 | 25.88 | 23.02 | 25.81 | |
| RTNModel=LLaMA-13B, Weight Precision=INT22024.02 | 28.16 | 27.15 | 26.09 | 25.53 | 26.73 | |
| RTNBackbone Model=LLaMA-30B, Quantization Precision=INT32024.02 | 27.9 | 43.64 | 31.42 | 23.34 | 31.58 | |
| AffineQuantModel=LLaMA-7B, Weight Precision=INT22024.02 | 27.9 | 49.58 | 51.85 | 24.15 | 38.37 | |
| RTNBackbone Model=LLaMA-7B, Quantization Precision=INT32024.02 | 27.47 | 45.45 | 45.46 | 24.94 | 35.83 | |
| OmniQuantModel=LLaMA-7B, Weight Precision=INT22024.02 | 27.22 | 49.2 | 50.65 | 23.74 | 37.7 | |
| OPTQModel=LLaMA-13B, Weight Precision=INT22024.02 | 27.22 | 25.76 | 25.67 | 25.05 | 25.93 | |
| RTNModel=LLaMA2-13B, Weight Precision=INT22024.02 | 27.22 | 25.04 | 25.58 | 24.69 | 25.63 | |
| OPTQModel=LLaMA-30B, Weight Precision=INT22024.02 | 27.13 | 26.6 | 26.12 | 23.56 | 25.85 | |
| OmniQuantModel=LLaMA2-13B, Weight Precision=INT22024.02 | 27.13 | 47.98 | 53.27 | 23.81 | 38.05 | |
| RTNModel=LLaMA-30B, Weight Precision=INT22024.02 | 27.05 | 26.39 | 25.87 | 25.48 | 26.2 | |
| OPTQModel=LLaMA2-13B, Weight Precision=INT22024.02 | 26.71 | 27.19 | 25.42 | 23.74 | 25.77 | |
| Z-FOLDModel=LLaMA2-7B, Weight Precision=INT22024.02 | 26.62 | 42.68 | 44.71 | 22.88 | 34.22 | |
| OPTQModel=LLaMA2-7B, Weight Precision=INT22024.02 | 26.37 | 26.09 | 25.11 | 25.1 | 25.67 | |
| RTNBackbone Model=LLaMA2-7B, Quantization Precision=INT32024.02 | 25.94 | 35.48 | 35.39 | 23.14 | 29.99 | |
| OmniQuantModel=LLaMA2-7B, Weight Precision=INT22024.02 | 25 | 38.8 | 42.97 | 23.03 | 32.45 |