Zero-shot Classification on 20-task CLIP evaluation suite (test)
76.61SUN AccuracyWUDI-Merging w/ E-PMQ
Evaluation Results
| Method | Links | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| WUDI-Merging w/ E-PMQMerging Method=WUDI-Merging, Quantization Method=E-PMQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 76.61 | 84.74 | 90.65 | 94.48 | 93.96 | 84.35 | 98.73 | 70.74 | 94.99 | 82.87 | 40.44 | 95.8 | 99.2 | 84.86 | 98.18 | 91.17 | 91.98 | 48.37 | 17.02 | 79.63 | 80.94 | |
| Simple Averaging w/ E-PMQMerging Method=Simple Averaging, Quantization Method=E-PMQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 72.12 | 82.74 | 83.48 | 79.67 | 94.5 | 71.14 | 97.93 | 65.16 | 92.31 | 80.59 | 40.9 | 95.31 | 98.83 | 84.11 | 98.16 | 92.68 | 90.27 | 49.53 | 13.02 | 70.4 | 77.64 | |
| Task Arithmetic w/ E-PMQMerging Method=Task Arithmetic, Quantization Method=E-PMQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 72.06 | 83.15 | 82.75 | 76.63 | 93.5 | 70.18 | 97.8 | 64.57 | 91.97 | 73.44 | 40.47 | 95.31 | 98.71 | 82.76 | 98.01 | 91.91 | 89.91 | 48.57 | 12.44 | 70.02 | 76.71 | |
| TIES-Merging w/ E-PMQMerging Method=TIES-Merging, Quantization Method=E-PMQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 71.99 | 82.89 | 84.06 | 80 | 94.51 | 71.1 | 97.9 | 65.59 | 92.36 | 78.2 | 40.47 | 95.39 | 98.73 | 83.88 | 98.19 | 92.56 | 90.35 | 47.65 | 13.13 | 70.07 | 77.45 | |
| Simple AveragingMerging Method=Simple Averaging, Quantization Method=Full Precision, Backbone=ViT-L/142026.05 | 70.7 | 77.7 | 76.4 | 75.3 | 69.5 | 62.1 | 93.7 | 57.7 | 80.8 | 73.6 | 52.7 | 94.2 | 99.2 | 81.7 | 97 | 90.7 | 77.4 | 16.1 | 10.4 | 66.1 | 71.1 | |
| Simple Averaging w/ GPTQMerging Method=Simple Averaging, Quantization Method=GPTQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 70.32 | 76.59 | 76.46 | 71.33 | 68.28 | 60.59 | 93.14 | 58.03 | 79.9 | 72.97 | 35.51 | 94.66 | 99.16 | 81.31 | 96.88 | 92.18 | 74.46 | 16.16 | 10.55 | 68.81 | 69.86 | |
| WUDI-MergingMerging Method=WUDI-Merging, Quantization Method=Full Precision, Backbone=ViT-L/142026.05 | 70.3 | 72.1 | 73.3 | 69.7 | 81.9 | 84.3 | 98.1 | 56.8 | 85.9 | 83.7 | 64.2 | 94.5 | 97.5 | 72.9 | 95.7 | 83.5 | 89.6 | 33.2 | 33.3 | 74.8 | 75.8 | |
| Simple Averaging w/ RTNMerging Method=Simple Averaging, Quantization Method=RTN, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 69.9 | 76.84 | 76.1 | 69.74 | 65.49 | 60.88 | 93.61 | 57.07 | 79.1 | 59.23 | 33.51 | 94.14 | 99.04 | 80.82 | 96.83 | 91.82 | 74.09 | 15.61 | 9.22 | 65.02 | 68.4 | |
| WUDI-Merging w/ GPTQMerging Method=WUDI-Merging, Quantization Method=GPTQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 69.77 | 70.43 | 73.49 | 63.78 | 81.69 | 73.64 | 98.11 | 57.55 | 84.88 | 76.66 | 39.2 | 94.03 | 97.29 | 72.45 | 95.53 | 82.37 | 89.98 | 30.9 | 13.39 | 78.64 | 72.19 | |
| WUDI-Merging w/ RTNMerging Method=WUDI-Merging, Quantization Method=RTN, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 69.19 | 68.75 | 71.27 | 62.07 | 80.09 | 72.41 | 98.05 | 55.59 | 83.67 | 74.26 | 38.92 | 94.17 | 97.25 | 71.49 | 95.32 | 80.92 | 89.64 | 30.03 | 13.35 | 78.36 | 71.24 | |
| WUDI-Merging w/ AWQMerging Method=WUDI-Merging, Quantization Method=AWQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 66.49 | 57.02 | 68.78 | 55.52 | 78.56 | 66.7 | 98.19 | 56.12 | 81.64 | 74.29 | 39.36 | 92.61 | 96.78 | 67.36 | 94.22 | 77.79 | 86.65 | 30.93 | 14.12 | 78.09 | 69.06 | |
| Simple Averaging w/ AWQMerging Method=Simple Averaging, Quantization Method=AWQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 66.07 | 63.61 | 62.3 | 47.15 | 48.4 | 46.14 | 86.97 | 52.71 | 73.28 | 54.83 | 36.15 | 92.31 | 96.98 | 70.23 | 93.34 | 83.94 | 63.21 | 12.17 | 10.84 | 65.07 | 61.29 | |
| TIES-MergingMerging Method=TIES-Merging, Quantization Method=Full Precision, Backbone=ViT-L/142026.05 | 64.4 | 56.6 | 49.1 | 42.1 | 67.2 | 56.8 | 95.3 | 46.2 | 64.9 | 78.3 | 54.9 | 91.3 | 95.7 | 62.9 | 92.9 | 70.5 | 82 | 19.9 | 10.9 | 56.9 | 63 | |
| TIES-Merging w/ GPTQMerging Method=TIES-Merging, Quantization Method=GPTQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 63.89 | 55.34 | 48.97 | 40.19 | 66.72 | 51.81 | 95.36 | 47.87 | 65.56 | 50.09 | 38.56 | 90.76 | 95.6 | 62.88 | 92.56 | 70.72 | 81.56 | 17.94 | 9.88 | 75.34 | 61.08 | |
| TIES-Merging w/ RTNMerging Method=TIES-Merging, Quantization Method=RTN, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 63.03 | 53.95 | 48.51 | 40.48 | 63.77 | 51.18 | 94.76 | 46.81 | 63.07 | 50.03 | 38.14 | 90.98 | 95.3 | 61.95 | 92 | 69.22 | 81.65 | 17.34 | 10.01 | 73.31 | 60.27 | |
| TIES-Merging w/ AWQMerging Method=TIES-Merging, Quantization Method=AWQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 60.02 | 44.1 | 44.32 | 33.59 | 60.25 | 45.9 | 95.88 | 45.37 | 64.42 | 50.08 | 39.37 | 87.95 | 94.46 | 60.35 | 90.73 | 64.43 | 77.84 | 19.09 | 11.07 | 71.61 | 58.04 | |
| Task Arithmetic w/ GPTQMerging Method=Task Arithmetic, Quantization Method=GPTQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 22.64 | 13.72 | 18.81 | 26.3 | 19.21 | 24.79 | 78.34 | 22.45 | 21.56 | 50.02 | 29.51 | 57.26 | 75.83 | 25.32 | 64.81 | 16.34 | 56.88 | 12.07 | 10.63 | 50.36 | 34.84 | |
| Task Arithmetic w/ RTNMerging Method=Task Arithmetic, Quantization Method=RTN, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 21.65 | 11.91 | 17.94 | 25 | 18.75 | 23.87 | 76.87 | 22.45 | 21.17 | 50.02 | 27.79 | 55.82 | 73.36 | 23.19 | 61.26 | 16.08 | 53.85 | 12.46 | 10.5 | 50.19 | 33.71 | |
| Task ArithmeticMerging Method=Task Arithmetic, Quantization Method=Full Precision, Backbone=ViT-L/142026.05 | 17.7 | 17.4 | 45.9 | 89.9 | 59.5 | 59.3 | 96.1 | 36.6 | 41.3 | 81.7 | 49.1 | 65.7 | 81.8 | 28 | 76.5 | 24.3 | 80.5 | 62.1 | 73.3 | 57.7 | 57.2 | |
| Task Arithmetic w/ AWQMerging Method=Task Arithmetic, Quantization Method=AWQ, Bit-width=4-bit, Backbone=ViT-L/14, Quantization Group Size=1282026.05 | 12.23 | 5.43 | 13.08 | 24.3 | 13.29 | 19.83 | 63.04 | 18.88 | 15.76 | 50.02 | 26.33 | 38.81 | 66.45 | 16.15 | 48.13 | 10.49 | 39.91 | 10.78 | 10.21 | 50.14 | 27.66 |