Python Program Synthesis on MBPP (test)
55.8AccuracyDAWN + AXON
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| DAWN + AXONBackbone=Dream-v0-Instruct-7B, Decoding strategy=DAWN + AXON2026.06 | 55.8 | 30.09 | 42.6 | |
| LocalLeap + AXONBackbone=Dream-v0-Instruct-7B, Decoding strategy=LocalLeap + AXON2026.06 | 55.6 | 28.47 | 41.4 | |
| Confidence + AXONBackbone=Dream-v0-Instruct-7B, Decoding strategy=Confidence + AXON2026.06 | 55.4 | 25.19 | 56 | |
| DAWNBackbone=Dream-v0-Instruct-7B, Decoding strategy=DAWN2026.06 | 55.4 | 29.76 | 42.8 | |
| ConfidenceBackbone=Dream-v0-Instruct-7B, Decoding strategy=Confidence2026.06 | 55.2 | 26.38 | 62.7 | |
| Confidence + AXONCVRBackbone=Dream-v0-Instruct-7B, Decoding strategy=Confidence + AXONCVR2026.06 | 55.2 | 24.98 | 55.1 | |
| LocalLeap + AXONCVRBackbone=Dream-v0-Instruct-7B, Decoding strategy=LocalLeap + AXONCVR2026.06 | 55 | 28.25 | 40.6 | |
| DAWN + AXONCVRBackbone=Dream-v0-Instruct-7B, Decoding strategy=DAWN + AXONCVR2026.06 | 55 | 29.89 | 41.8 | |
| LocalLeapBackbone=Dream-v0-Instruct-7B, Decoding strategy=LocalLeap2026.06 | 54.6 | 28.69 | 44.7 | |
| OriginalBackbone=Dream-v0-Instruct-7B, Decoding strategy=Original2026.06 | 54.2 | 5.42 | 256 | |
| Confidence + AXONBackbone=LLaDA-1.5, Decoding strategy=Confidence + AXON2026.06 | 39.2 | 14.39 | 42.2 | |
| OriginalBackbone=LLaDA-1.5, Decoding strategy=Original2026.06 | 38.8 | 2.39 | 256 | |
| LocalLeapBackbone=LLaDA-1.5, Decoding strategy=LocalLeap2026.06 | 38.8 | 18.5 | 32.5 | |
| Confidence + AXONCVRBackbone=LLaDA-1.5, Decoding strategy=Confidence + AXONCVR2026.06 | 38.2 | 14.52 | 42.1 | |
| LocalLeap + AXONBackbone=LLaDA-1.5, Decoding strategy=LocalLeap + AXON2026.06 | 38.2 | 16.77 | 35.3 | |
| ConfidenceBackbone=LLaDA-1.5, Decoding strategy=Confidence2026.06 | 38 | 14.11 | 42.5 | |
| DAWNBackbone=LLaDA-1.5, Decoding strategy=DAWN2026.06 | 37.8 | 19.37 | 30.7 | |
| LocalLeap + AXONCVRBackbone=LLaDA-1.5, Decoding strategy=LocalLeap + AXONCVR2026.06 | 37.6 | 17.13 | 34.6 | |
| DAWN + AXONBackbone=LLaDA-1.5, Decoding strategy=DAWN + AXON2026.06 | 37.2 | 20.97 | 29.3 | |
| DAWN + AXONCVRBackbone=LLaDA-1.5, Decoding strategy=DAWN + AXONCVR2026.06 | 37.2 | 21.49 | 28.3 | |
| DAWN + AXONBackbone=LLaDA-8B-Instruct, Decoding strategy=DAWN + AXON2026.06 | 32.4 | 31.69 | 44.4 | |
| Confidence + AXONBackbone=LLaDA-8B-Instruct, Decoding strategy=Confidence + AXON2026.06 | 32 | 23.51 | 62.2 | |
| Confidence + AXONCVRBackbone=LLaDA-8B-Instruct, Decoding strategy=Confidence + AXONCVR2026.06 | 31.4 | 23.69 | 61.6 | |
| LocalLeap + AXONBackbone=LLaDA-8B-Instruct, Decoding strategy=LocalLeap + AXON2026.06 | 31.4 | 28.05 | 51.5 | |
| DAWNBackbone=LLaDA-8B-Instruct, Decoding strategy=DAWN2026.06 | 30.8 | 31.19 | 50 | |
| DAWN + AXONCVRBackbone=LLaDA-8B-Instruct, Decoding strategy=DAWN + AXONCVR2026.06 | 30.6 | 32.35 | 42.4 | |
| LocalLeap + AXONCVRBackbone=LLaDA-8B-Instruct, Decoding strategy=LocalLeap + AXONCVR2026.06 | 30.2 | 28.61 | 50.6 | |
| ConfidenceBackbone=LLaDA-8B-Instruct, Decoding strategy=Confidence2026.06 | 29.6 | 23.69 | 69.8 | |
| OriginalBackbone=LLaDA-8B-Instruct, Decoding strategy=Original2026.06 | 29.4 | 6.48 | 256 | |
| LocalLeapBackbone=LLaDA-8B-Instruct, Decoding strategy=LocalLeap2026.06 | 29.4 | 29.84 | 55 |