Program Synthesis on PSB1
89Checksum CorrectnessCopilot
Evaluation results
| Method | Links | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Copilot2025.11 | 89 | 73 | 70 | 98 | 88 | 11 | 72 | 84 | 61 | 79 | 70 | 99 | 93 | 54 | 87 | 35 | 51 | 66 | 60 | 90 | 55 | 96 | 92 | 87 | 1 | — | 0 | 12 | 22 | 23 | 26 | — | |
| Copilot2025.11 | 89 | 73 | 70 | 98 | 88 | 11 | 72 | 84 | 61 | 79 | 70 | 99 | 93 | 54 | 87 | 35 | 51 | 66 | 60 | 90 | 55 | 96 | 92 | 87 | 1 | — | 0 | 12 | 22 | 23 | 26 | 9 | |
| Origami AC/DC2025.11 | 70 | 0 | 97 | 100 | 63 | 38 | 97 | 100 | 93 | 100 | 100 | 100 | 100 | 0 | 20 | 100 | 67 | 100 | 100 | 30 | 63 | 80 | 100 | 87 | 0 | — | 11 | 16 | 20 | 21 | 22 | 0 | |
| Origami2025.11 | 0 | 0 | 90 | 40 | 3 | 3 | 90 | 100 | 70 | 97 | 93 | 87 | 100 | 0 | 3 | 100 | 53 | 100 | 97 | 0 | 73 | 7 | 0 | 20 | 0 | — | 4 | 10 | 13 | 14 | 19 | — |