Instruction Following on Vicuna benchmark zero-shot
119.4Pairwise Score (ChatGPT vs Sys)GPT-4
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| GPT-4Zero-shot=true2023.05 | 119.4 | 110.1 | 114.5 | 2.6 | |
| GuanacoParams=65B, Model bits=4-bit, Memory=41 GB, Zero-shot=true2023.05 | 96.7 | 101.9 | 99.3 | 4.4 | |
| GuanacoParams=33B, Model bits=4-bit, Memory=21 GB, Zero-shot=true2023.05 | 96.5 | 99.2 | 97.8 | 4.4 | |
| BardZero-shot=true2023.05 | 93.2 | 96.4 | 94.8 | 4.1 | |
| Open AssistantParams=33B, Model bits=16-bit, Memory=66 GB, Zero-shot=true2023.05 | 91.2 | 98.7 | 94.9 | 4.5 | |
| VicunaParams=13B, Model bits=16-bit, Memory=26 GB, Zero-shot=true2023.05 | 91.2 | 98.7 | 94.9 | 4.5 | |
| GuanacoParams=13B, Model bits=4-bit, Memory=10 GB, Zero-shot=true2023.05 | 87.3 | 93.4 | 90.4 | 5.2 | |
| GuanacoParams=7B, Model bits=4-bit, Memory=5 GB, Zero-shot=true2023.05 | 84.1 | 89.8 | 87 | 5.4 | |
| AlpacaParams=33B, Model bits=4-bit, Memory=21 GB, Zero-shot=true2023.05 | 67.2 | 79.7 | 73.6 | 4.2 | |
| AlpacaParams=13B, Model bits=4-bit, Memory=10 GB, Zero-shot=true2023.05 | 63.8 | 76.7 | 69.4 | 4.2 | |
| AlpacaParams=65B, Model bits=4-bit, Memory=41 GB, Zero-shot=true2023.05 | 63 | 77.9 | 70.7 | 4.3 | |
| AlpacaParams=7B, Model bits=4-bit, Memory=5 GB, Zero-shot=true2023.05 | 57.3 | 71.2 | 64.4 | 5 | |
| HH-RLHFParams=13B, Model bits=4-bit, Memory=10 GB, Zero-shot=true2023.05 | 55.5 | 69.1 | 62.5 | 4.7 | |
| Unnatural Instr.Params=13B, Model bits=4-bit, Memory=10 GB, Zero-shot=true2023.05 | 50.6 | 69.8 | 60.5 | 4.2 | |
| Chip2Params=13B, Model bits=4-bit, Memory=10 GB, Zero-shot=true2023.05 | 49.2 | 69.3 | 59.5 | 4.7 | |
| LongformParams=13B, Model bits=4-bit, Memory=10 GB, Zero-shot=true2023.05 | 44.9 | 62 | 53.6 | 5.2 | |
| Self-InstructParams=13B, Model bits=4-bit, Memory=10 GB, Zero-shot=true2023.05 | 38 | 60.5 | 49.1 | 4.6 | |
| FLAN v2Params=65B, Model bits=4-bit, Memory=41 GB, Zero-shot=true2023.05 | 37 | 59.6 | 48.4 | 4.6 | |
| FLAN v2Params=7B, Model bits=4-bit, Memory=5 GB, Zero-shot=true2023.05 | 33.3 | 56.1 | 44.8 | 4 | |
| FLAN v2Params=13B, Model bits=4-bit, Memory=10 GB, Zero-shot=true2023.05 | 32.4 | 61.2 | 47 | 3.6 | |
| FLAN v2Params=33B, Model bits=4-bit, Memory=21 GB, Zero-shot=true2023.05 | 26.3 | 49.7 | 38 | 3.9 |