Code Generation on MBPP (Acc@t1, Acc@t2, Δ(t1,t2))
66.2Accuracy @ t1Prompt based
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Prompt basedBackbone=Meta-Llama-3-70B-Instruct, Variant=w/o reflection2025.05 | 66.2 | — | — | |
| Prompt basedBackbone=Meta-Llama-3-70B-Instruct, Variant=w/ reflection (variant 1)2025.05 | 66.2 | 71 | 4.8 | |
| Prompt basedBackbone=Meta-Llama-3-70B-Instruct, Variant=w/ reflection (variant 2)2025.05 | 66.2 | 73 | 6.8 | |
| Prompt basedBackbone=Gemma-2-27B-it, Configuration=w/o reflection2025.05 | 65.4 | 69.4 | 4 | |
| Prompt basedBackbone=Gemma-2-27B-it, Configuration=w/ reflection2025.05 | 65.4 | 69.2 | 3.8 | |
| Prompt basedBackbone=Mistral-22B-Small-Instruct, Variant=w/o reflection2025.05 | 63 | — | — | |
| Prompt basedBackbone=Mistral-22B-Small-Instruct, Variant=w/ reflection (variant 1)2025.05 | 63 | 68 | 5 | |
| Prompt basedBackbone=Mistral-22B-Small-Instruct, Variant=w/ reflection (variant 2)2025.05 | 63 | 68 | 5 | |
| Prompt basedBackbone=Gemma-2-9B-it, Configuration=w/o reflection2025.05 | 54.4 | 59.2 | 4.8 | |
| Prompt basedBackbone=Gemma-2-9B-it, Configuration=w/ reflection2025.05 | 54.4 | 59.6 | 5.2 | |
| ReflectEvoBackbone=Gemma-2-9B-it, Configuration=one-stage w/ D+2025.05 | 54.4 | 57.8 | 3.4 | |
| ReflectEvoBackbone=Gemma-2-9B-it, Configuration=two-stage w/ D+2025.05 | 54.4 | 56.8 | 2.4 | |
| ReflectEvoBackbone=Gemma-2-9B-it, Configuration=w/ D-2025.05 | 54.4 | 58.6 | 4.2 | |
| ReflectEvoBackbone=Gemma-2-9B-it, Configuration=w/ Dpref2025.05 | 54.4 | 59.4 | 5 | |
| SFT basedBackbone=Gemma-2-9B-it, Configuration=w/ SFT qa pairs2025.05 | 37 | — | — | |
| SFT basedBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ SFT qa pairs2025.05 | 31.2 | — | — | |
| Prompt basedBackbone=Meta-Llama-3-8B-Instruct, Variant=w/o reflection2025.05 | 28.4 | — | — | |
| Prompt basedBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ reflection2025.05 | 28.4 | 44 | 15.6 | |
| ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=one-stage w/ D+2025.05 | 28.4 | 29.6 | 1.2 | |
| ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=two-stage w/ D+2025.05 | 28.4 | 42.4 | 14 | |
| ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ D+/-2025.05 | 28.4 | 48.8 | 20.4 | |
| ReflectEvoBackbone=Meta-Llama-3-8B-Instruct, Variant=w/ Dpref2025.05 | 28.4 | 47.4 | 19 | |
| Prompt basedBackbone=Mistral-7B-Instruct-v0.2, Variant=w/o reflection2025.05 | 20.4 | — | — | |
| Prompt basedBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ reflection2025.05 | 20.4 | 23 | 2.6 | |
| ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=one-stage w/ D+2025.05 | 20.4 | 24 | 3.6 | |
| ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=two-stage w/ D+2025.05 | 20.4 | 20.8 | 0.4 | |
| ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ D+/-2025.05 | 20.4 | 23.2 | 2.8 | |
| ReflectEvoBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ Dpref2025.05 | 20.4 | 22.6 | 2.2 | |
| SFT basedBackbone=Mistral-7B-Instruct-v0.2, Variant=w/ SFT qa pairs2025.05 | 17 | — | — |