DRC Script Synthesis on Rule2DRC (test)
72.1Success RateOracle@20
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Oracle@20Base Model=GPT-OSS-120B, Best-of-N=202026.05 | 72.1 | 3.7 | |
| Oracle@15Base Model=GPT-OSS-120B, Best-of-N=152026.05 | 69 | 4.9 | |
| SplitTesterBase Model=GPT-OSS-120B, Best-of-N=20, Test Layouts=162026.05 | 63.8 | 4.1 | |
| Oracle@10Base Model=GPT-OSS-120B, Best-of-N=102026.05 | 63 | 8.5 | |
| CodeMonkeyBase Model=GPT-OSS-120B, Best-of-N=20, Test Layouts=162026.05 | 62.7 | 4.2 | |
| SplitTesterBase Model=GPT-OSS-120B, Best-of-N=15, Test Layouts=162026.05 | 61.1 | 5.3 | |
| CodeMonkeyBase Model=GPT-OSS-120B, Best-of-N=15, Test Layouts=162026.05 | 60.2 | 5.3 | |
| Generated Tests (8 Tests)Base Model=GPT-OSS-120B, Best-of-N=20, Test Layouts=82026.05 | 59.4 | 4.2 | |
| S*Base Model=GPT-OSS-120B, Best-of-N=20, Test Layouts=162026.05 | 59.3 | 4.2 | |
| SplitTesterBase Model=GPT-OSS-120B, Best-of-N=10, Test Layouts=162026.05 | 58 | 9.1 | |
| Generated Tests (16 Tests)Base Model=GPT-OSS-120B, Best-of-N=20, Test Layouts=162026.05 | 57.9 | 4.2 | |
| CodeMonkeyBase Model=GPT-OSS-120B, Best-of-N=10, Test Layouts=162026.05 | 56.9 | 9.1 | |
| Generated Tests (8 Tests)Base Model=GPT-OSS-120B, Best-of-N=15, Test Layouts=82026.05 | 56.4 | 5.4 | |
| S*Base Model=GPT-OSS-120B, Best-of-N=15, Test Layouts=162026.05 | 56.3 | 5.4 | |
| Generated Tests (16 Tests)Base Model=GPT-OSS-120B, Best-of-N=15, Test Layouts=162026.05 | 55.6 | 5.4 | |
| Generated Tests (8 Tests)Base Model=GPT-OSS-120B, Best-of-N=10, Test Layouts=82026.05 | 54.5 | 9.1 | |
| S*Base Model=GPT-OSS-120B, Best-of-N=10, Test Layouts=162026.05 | 54.3 | 9.1 | |
| Generated Tests (16 Tests)Base Model=GPT-OSS-120B, Best-of-N=10, Test Layouts=162026.05 | 53.9 | 9.1 | |
| Oracle@20Base Model=GPT-OSS-20B, Best-of-N=202026.05 | 53.8 | 11.6 | |
| Oracle@15Base Model=GPT-OSS-20B, Best-of-N=152026.05 | 50.1 | 15 | |
| SplitTesterBase Model=GPT-OSS-20B, Best-of-N=20, Test Layouts=162026.05 | 44.4 | 12.6 | |
| Oracle@10Base Model=GPT-OSS-20B, Best-of-N=102026.05 | 44.1 | 19.7 | |
| SplitTesterBase Model=GPT-OSS-20B, Best-of-N=15, Test Layouts=162026.05 | 41.9 | 16 | |
| CodeMonkeyBase Model=GPT-OSS-20B, Best-of-N=20, Test Layouts=162026.05 | 41.1 | 14.1 | |
| CodeMonkeyBase Model=GPT-OSS-20B, Best-of-N=15, Test Layouts=162026.05 | 40.2 | 17.2 | |
| Generated Tests (8 Tests)Base Model=GPT-OSS-20B, Best-of-N=20, Test Layouts=82026.05 | 39 | 14 | |
| SplitTesterBase Model=GPT-OSS-20B, Best-of-N=10, Test Layouts=162026.05 | 38.7 | 20.5 | |
| Generated Tests (16 Tests)Base Model=GPT-OSS-20B, Best-of-N=20, Test Layouts=162026.05 | 38.7 | 14.2 | |
| S*Base Model=GPT-OSS-20B, Best-of-N=20, Test Layouts=162026.05 | 38.2 | 14.9 | |
| Generated Tests (8 Tests)Base Model=GPT-OSS-20B, Best-of-N=15, Test Layouts=82026.05 | 38 | 17.2 | |
| LLM-as-a-JudgeBase Model=GPT-OSS-120B, Best-of-N=15, Test Layouts=–2026.05 | 37.8 | 46.6 | |
| S*Base Model=GPT-OSS-20B, Best-of-N=15, Test Layouts=162026.05 | 37.7 | 17.3 | |
| LLM-as-a-JudgeBase Model=GPT-OSS-120B, Best-of-N=10, Test Layouts=–2026.05 | 37.6 | 47.6 | |
| LLM-as-a-JudgeBase Model=GPT-OSS-120B, Best-of-N=20, Test Layouts=–2026.05 | 37.6 | 47.5 | |
| Generated Tests (16 Tests)Base Model=GPT-OSS-20B, Best-of-N=15, Test Layouts=162026.05 | 37.3 | 17.4 | |
| CodeMonkeyBase Model=GPT-OSS-20B, Best-of-N=10, Test Layouts=162026.05 | 36.6 | 21.8 | |
| S*Base Model=GPT-OSS-20B, Best-of-N=10, Test Layouts=162026.05 | 35.8 | 21.8 | |
| Generated Tests (8 Tests)Base Model=GPT-OSS-20B, Best-of-N=10, Test Layouts=82026.05 | 35.3 | 21.8 | |
| Generated Tests (16 Tests)Base Model=GPT-OSS-20B, Best-of-N=10, Test Layouts=162026.05 | 34.7 | 21.9 | |
| Sample-1Base Model=GPT-OSS-120B, Best-of-N=12026.05 | 32.5 | 48.5 | |
| LLM-as-a-JudgeBase Model=GPT-OSS-20B, Best-of-N=20, Test Layouts=–2026.05 | 28 | 54.9 | |
| LLM-as-a-JudgeBase Model=GPT-OSS-20B, Best-of-N=15, Test Layouts=–2026.05 | 26.2 | 57.5 | |
| LLM-as-a-JudgeBase Model=GPT-OSS-20B, Best-of-N=10, Test Layouts=–2026.05 | 25.1 | 56.6 | |
| Oracle@20Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=202026.05 | 23.7 | 29.5 | |
| Oracle@15Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=152026.05 | 22.6 | 32 | |
| Oracle@10Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=102026.05 | 21.4 | 34.8 | |
| SplitTesterBase Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=15, Test Layouts=162026.05 | 18 | 37.9 | |
| SplitTesterBase Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=20, Test Layouts=162026.05 | 18 | 35.1 | |
| SplitTesterBase Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=10, Test Layouts=162026.05 | 17.5 | 39.8 | |
| Generated Tests (8 Tests)Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=20, Test Layouts=82026.05 | 17.3 | 38.5 | |
| CodeMonkeyBase Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=15, Test Layouts=162026.05 | 17.2 | 40.2 | |
| CodeMonkeyBase Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=20, Test Layouts=162026.05 | 17.2 | 38.6 | |
| Generated Tests (8 Tests)Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=15, Test Layouts=82026.05 | 17.1 | 40.2 | |
| Sample-1Base Model=GPT-OSS-20B, Best-of-N=12026.05 | 16.9 | 66.9 | |
| Generated Tests (16 Tests)Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=15, Test Layouts=162026.05 | 16.9 | 40.1 | |
| CodeMonkeyBase Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=10, Test Layouts=162026.05 | 16.8 | 42.4 | |
| Generated Tests (16 Tests)Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=20, Test Layouts=162026.05 | 16.8 | 39 | |
| Generated Tests (8 Tests)Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=10, Test Layouts=82026.05 | 16.4 | 42.3 | |
| Generated Tests (16 Tests)Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=10, Test Layouts=162026.05 | 16.2 | 42.6 | |
| S*Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=15, Test Layouts=162026.05 | 16.2 | 40.2 | |
| LLM-as-a-JudgeBase Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=20, Test Layouts=–2026.05 | 16 | 65.2 | |
| S*Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=10, Test Layouts=162026.05 | 15.8 | 42.4 | |
| LLM-as-a-JudgeBase Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=10, Test Layouts=–2026.05 | 15.7 | 64.2 | |
| LLM-as-a-JudgeBase Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=15, Test Layouts=–2026.05 | 15.7 | 63.6 | |
| S*Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=20, Test Layouts=162026.05 | 15.7 | 38.6 | |
| Sample-1Base Model=Qwen3-30B-A3B-Instruct-2507, Best-of-N=12026.05 | 14.1 | 61.9 |