Zero-shot Performance on LMEval Harness Subset (HellaSwag, BoolQ, WinoGrande, PiQA, ARC)
75.46Mean AccuracyGIFT-SW
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| GIFT-SWBits=4 bit, Backbone=LLaMA3-8b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 75.46 | — | — | — | — | — | — | |
| STEBits=4 bit, Backbone=LLaMA2-13b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 75.29 | — | — | — | — | — | — | |
| STEBits=4 bit, Backbone=LLaMA3-8b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 74.84 | — | — | — | — | — | — | |
| GIFT-SWBits=4 bit, Backbone=LLaMA2-13b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 74.5 | — | — | — | — | — | — | |
| STEBits=3 bit, Backbone=LLaMA2-13b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 74.37 | — | — | — | — | — | — | |
| GIFT-SWBits=3 bit, Backbone=LLaMA2-13b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 74.34 | — | — | — | — | — | — | |
| FP16Model=L3-8B, Bits=16, Group size=1282026.02 | 74.31 | 73.32 | 80.9 | 79.23 | 81.31 | 77.74 | 53.33 | |
| QUIK + LORABits=4 bit, Backbone=LLaMA3-8b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 74.27 | — | — | — | — | — | — | |
| GIFT-SWBits=3 bit, Backbone=LLaMA3-8b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 73.27 | — | — | — | — | — | — | |
| SNRQ-SModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=498.72026.02 | 73.26 | 73.24 | 80.23 | 77.88 | 80.18 | 76.76 | 51.26 | |
| GPTAQModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=703.22026.02 | 73.23 | 73.69 | 80.34 | 78.16 | 79.99 | 76.48 | 50.7 | |
| FP16Model=L2-13B, Bits=16, Group size=1282026.02 | 73.23 | 72.3 | 80.52 | 79.38 | 80.58 | 77.53 | 49.06 | |
| SNRQ-CModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=495.22026.02 | 73.08 | 73.09 | 80.27 | 77.83 | 79.94 | 76.62 | 50.72 | |
| LDLQModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=425.12026.02 | 73.05 | 73.16 | 80.12 | 78.14 | 79.58 | 76.67 | 50.61 | |
| GPTQModel=L3-8B, Bits=4, Group size=128, Q.Time (s)=4892026.02 | 72.98 | 73.51 | 80.19 | 77.97 | 80.28 | 76 | 49.95 | |
| SNRQ-CModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=726.12026.02 | 72.8 | 72.38 | 80.11 | 78.65 | 79.86 | 76.95 | 48.82 | |
| LDLQModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=623.92026.02 | 72.79 | 72.58 | 79.87 | 78.75 | 79.98 | 76.79 | 48.75 | |
| GPTAQModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=1019.22026.02 | 72.71 | 71.87 | 79.83 | 78.8 | 80.09 | 76.88 | 48.82 | |
| SNRQ-SModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=709.42026.02 | 72.67 | 72.06 | 79.99 | 78.75 | 79.79 | 76.69 | 48.75 | |
| GPTQModel=L2-13B, Bits=4, Group size=128, Q.Time (s)=727.72026.02 | 72.62 | 72.06 | 79.91 | 78.78 | 79.42 | 76.83 | 48.72 | |
| GIFT-SWBits=4 bit, Backbone=LLaMA2-7b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 72.53 | — | — | — | — | — | — | |
| STEBits=4 bit, Backbone=LLaMA2-7b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 72.43 | — | — | — | — | — | — | |
| QUIK + LORABits=3 bit, Backbone=LLaMA3-8b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 71.65 | — | — | — | — | — | — | |
| QUIK + LORABits=3 bit, Backbone=LLaMA2-13b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 71.3 | — | — | — | — | — | — | |
| SNRQ-SModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=724.22026.02 | 71.11 | 71.19 | 79.07 | 75.95 | 78.85 | 74.94 | 46.63 | |
| QUIK + LORABits=4 bit, Backbone=LLaMA2-13b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 71.08 | — | — | — | — | — | — | |
| SNRQ-CModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=749.62026.02 | 71.06 | 70.69 | 79.44 | 76.03 | 78.33 | 75.18 | 46.72 | |
| GIFT-SWBits=3 bit, Backbone=LLaMA2-7b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 71 | — | — | — | — | — | — | |
| GPTAQModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=10122026.02 | 71 | 70.83 | 79.36 | 76.36 | 79.05 | 74.11 | 46.42 | |
| LDLQModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=719.22026.02 | 70.97 | 70.26 | 79.11 | 76.24 | 78.94 | 74.8 | 46.47 | |
| GPTQModel=L2-13B, Bits=3, Group size=128, Q.Time (s)=820.52026.02 | 70.93 | 70.54 | 79.27 | 76.32 | 79.08 | 73.91 | 46.45 | |
| FP16Model=L2-7B, Bits=16, Group size=1282026.02 | 70.47 | 69.06 | 79.11 | 75.98 | 77.74 | 74.62 | 46.33 | |
| STEBits=3 bit, Backbone=LLaMA3-8b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 70.24 | — | — | — | — | — | — | |
| STEBits=3 bit, Backbone=LLaMA2-7b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 69.82 | — | — | — | — | — | — | |
| SNRQ-SModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=405.62026.02 | 69.65 | 69.2 | 78.58 | 75.14 | 76.41 | 73.47 | 45.07 | |
| GPTQModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=481.62026.02 | 69.58 | 69.14 | 78.4 | 75.11 | 77.25 | 73.13 | 44.47 | |
| LDLQModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=404.82026.02 | 69.54 | 69.69 | 78.44 | 75.21 | 76.77 | 73.04 | 44.11 | |
| SNRQ-CModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=450.92026.02 | 69.49 | 68.83 | 78.56 | 75.07 | 76.2 | 73.49 | 44.81 | |
| GPTAQModel=L2-7B, Bits=4, Group size=128, Q.Time (s)=610.52026.02 | 69.43 | 69.72 | 78.52 | 75.16 | 75.32 | 73.3 | 44.57 | |
| SNRQ-SModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=484.72026.02 | 69.33 | 70.64 | 77.56 | 73.28 | 77.55 | 71.62 | 45.36 | |
| GIFT-SWBits=2 bit, Backbone=LLaMA2-13b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 67.61 | — | — | — | — | — | — | |
| SNRQ-CModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=492.32026.02 | 67.6 | 70.48 | 76.43 | 73.06 | 76.36 | 67.54 | 41.76 | |
| SNRQ-SModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=417.32026.02 | 67.34 | 68.02 | 77.4 | 71.69 | 75.07 | 70.18 | 41.37 | |
| GPTQModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=547.52026.02 | 66.82 | 69.14 | 75.52 | 73.07 | 73.76 | 67.1 | 42.32 | |
| SNRQ-CModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=407.42026.02 | 66.82 | 67.48 | 77.68 | 71.55 | 72.94 | 70.02 | 41.26 | |
| LDLQModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=427.82026.02 | 66.5 | 66.84 | 77.02 | 71.73 | 74.12 | 68.49 | 40.79 | |
| GPTAQModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=590.42026.02 | 66.34 | 66.71 | 77.27 | 72 | 73.1 | 68.11 | 40.84 | |
| GPTQModel=L2-7B, Bits=3, Group size=128, Q.Time (s)=467.92026.02 | 66.14 | 65.97 | 76.71 | 71.88 | 73.11 | 67.93 | 41.21 | |
| GPTAQModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=7152026.02 | 65.77 | 69.66 | 74.7 | 73.89 | 74.75 | 61.89 | 39.76 | |
| LDLQModel=L3-8B, Bits=3, Group size=128, Q.Time (s)=472.52026.02 | 65.2 | 69.82 | 74.97 | 70.81 | 71.27 | 63.47 | 40.84 | |
| QUIK + LORABits=4 bit, Backbone=LLaMA2-7b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 63.99 | — | — | — | — | — | — | |
| QUIK + LORABits=3 bit, Backbone=LLaMA2-7b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 62.91 | — | — | — | — | — | — | |
| STEBits=2 bit, Backbone=LLaMA2-13b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 62.19 | — | — | — | — | — | — | |
| GIFT-SWBits=2 bit, Backbone=LLaMA2-7b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 61.09 | — | — | — | — | — | — | |
| GIFT-SWBits=2 bit, Backbone=LLaMA3-8b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 58.89 | — | — | — | — | — | — | |
| STEBits=2 bit, Backbone=LLaMA2-7b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 58.2 | — | — | — | — | — | — | |
| QUIK + LORABits=2 bit, Backbone=LLaMA3-8b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 53.8 | — | — | — | — | — | — | |
| STEBits=2 bit, Backbone=LLaMA3-8b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 48.96 | — | — | — | — | — | — | |
| QUIK + LORABits=2 bit, Backbone=LLaMA2-13b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 47.14 | — | — | — | — | — | — | |
| QUIK + LORABits=2 bit, Backbone=LLaMA2-7b, Fine-tuning dataset=TÜLU-V2-mix2024.08 | 41.44 | — | — | — | — | — | — |