Natural Language Understanding on Ag_news, Subj, MR, Boolq, RTE (test)
71.6Ag_news AccuracyDecoQuant
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| DecoQuantBackbone=OPT-6.7B, Bits=4, ICL=2-shot2024.05 | 71.6 | 73.6 | 87 | 68.2 | 53.4 | 71.2 | |
| FP16Backbone=OPT-6.7B, Bits=16, ICL=2-shot2024.05 | 71 | 74 | 89.9 | 65.7 | 54.2 | 71 | |
| FP16Backbone=OPT-6.7B, Bits=16, ICL=0-shot2024.05 | 70.9 | 61.4 | 64.3 | 63.5 | 60.3 | 64.1 | |
| FP16Backbone=OPT-1.3B, Bits=16, ICL=10-shot2024.05 | 70 | 64.4 | 84 | 64.7 | 50.2 | 66.7 | |
| RTNBackbone=OPT-6.7B, Bits=4, ICL=2-shot2024.05 | 68.7 | 66.1 | 81.1 | 67.5 | 53.8 | 69.2 | |
| FP16Backbone=OPT-1.3B, Bits=16, ICL=2-shot2024.05 | 64.2 | 55.1 | 86.1 | 56.9 | 45.1 | 61.5 | |
| RTNBackbone=OPT-1.3B, Bits=4, ICL=10-shot2024.05 | 63.6 | 51.7 | 83.7 | 63 | 48.7 | 62.1 | |
| DecoQuantBackbone=OPT-1.3B, Bits=4, ICL=10-shot2024.05 | 62.6 | 69.7 | 85.6 | 63 | 48.4 | 65.9 | |
| DecoQuantBackbone=OPT-1.3B, Bits=4, ICL=2-shot2024.05 | 62.4 | 55.8 | 87 | 52.2 | 46.9 | 60.9 | |
| RTNBackbone=OPT-1.3B, Bits=4, ICL=2-shot2024.05 | 61.7 | 63.1 | 81.1 | 41.1 | 45.1 | 58.4 | |
| FP16Backbone=OPT-1.3B, Bits=16, ICL=0-shot2024.05 | 58 | 62.9 | 79.5 | 60.5 | 52.7 | 62.7 | |
| DecoQuantBackbone=OPT-6.7B, Bits=4, ICL=10-shot2024.05 | 54.6 | 92.4 | 85.6 | 62.6 | 51.3 | 70 | |
| FP16Backbone=OPT-6.7B, Bits=16, ICL=10-shot2024.05 | 53.3 | 89.8 | 86.8 | 65.7 | 57 | 70.5 | |
| RTNBackbone=OPT-6.7B, Bits=4, ICL=10-shot2024.05 | 53.1 | 76.8 | 83.7 | 64.7 | 54.5 | 67.2 | |
| RTNBackbone=OPT-6.7B, Bits=2, ICL=10-shot2024.05 | 45.7 | 51.7 | 52.7 | 47.5 | 50.2 | 49 | |
| DecoQuantBackbone=OPT-6.7B, Bits=2, ICL=10-shot2024.05 | 44.9 | 48.8 | 54.1 | 60.1 | 53.4 | 51.8 | |
| DecoQuantBackbone=OPT-1.3B, Bits=2, ICL=10-shot2024.05 | 42.4 | 65.6 | 54.1 | 43.4 | 52.7 | 51.6 | |
| DecoQuantBackbone=OPT-1.3B, Bits=2, ICL=2-shot2024.05 | 40.4 | 56 | 52.1 | 49.7 | 52.3 | 51.6 | |
| RTNBackbone=OPT-1.3B, Bits=2, ICL=10-shot2024.05 | 37.6 | 53.7 | 52.7 | 39 | 49.5 | 51.4 | |
| RTNBackbone=OPT-1.3B, Bits=2, ICL=2-shot2024.05 | 33 | 51.7 | 55 | 41.8 | 53.1 | 46.9 | |
| DecoQuantBackbone=OPT-6.7B, Bits=2, ICL=2-shot2024.05 | 32 | 55.1 | 52.1 | 61.5 | 53.4 | 53.1 | |
| RTNBackbone=OPT-6.7B, Bits=2, ICL=2-shot2024.05 | 29.3 | 51.7 | 55 | 38 | 52 | 44.2 |