Logical Reasoning on Synlogic (Pass@1 to Pass@128)
34.6Pass@1CaSP
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| CaSPModel=Qwen2.5-7B2025.10 | 34.6 | 38.3 | 41.8 | 45.3 | 48.1 | 50.7 | 53.1 | 55.1 | |
| GRPOModel=Qwen2.5-7B2025.10 | 34 | 36.7 | 39.1 | 41.1 | 43.1 | 45 | 47 | 48.6 | |
| PSRModel=Qwen2.5-7B2025.10 | 27.3 | 29.3 | 31.7 | 34.3 | 36.9 | 39.4 | 41.6 | 43.6 | |
| Base ModelModel=Qwen2.5-7B2025.10 | 3.1 | 4.9 | 7.5 | 11.1 | 15.4 | 20 | 24.5 | 28.7 | |
| W-REINFORCEModel=Qwen2.5-7B2025.10 | 0.8 | 1.3 | 1.8 | 2.4 | 2.9 | 3.4 | 3.8 | 3.9 |