Programming on HumanEval (test)
79.5ACC1o1-mini
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| o1-miniModel Family=ChatGPT2024.12 | 79.5 | 4.3 | 14.8 | |
| 4oModel Family=ChatGPT2024.12 | 72.6 | 6.8 | 21.9 | |
| GPT-4number_of_shots=0-shot2023.03 | 67 | — | — | |
| CodeT + GPT-3.5benchmark-specific tuning=true2023.03 | 65.8 | — | — | |
| FP16Bits=16-bit, Backbone=WizardCoder-7B2023.11 | 57.31 | — | — | |
| AFPQ (NF3-asym)Bits=3-bit, Quantization Format=NF3, Symmetry=Asymmetric, Framework=AWQ, Group-size=64, Backbone=WizardCoder-7B2023.11 | 52.43 | — | — | |
| 3.5-turboModel Family=ChatGPT2024.12 | 50.9 | 10.6 | 28.3 | |
| GPT-3.5number_of_shots=0-shot2023.03 | 48.1 | — | — | |
| AWQ (INT3)Bits=3-bit, Quantization Format=INT3, Framework=AWQ, Group-size=64, Backbone=WizardCoder-7B2023.11 | 47.56 | — | — | |
| AWQ (NF3-sym)Bits=3-bit, Quantization Format=NF3, Symmetry=Symmetric, Framework=AWQ, Group-size=64, Backbone=WizardCoder-7B2023.11 | 45.12 | — | — | |
| PaLMnumber_of_shots=0-shot2023.03 | 26.2 | — | — |