Long-horizon cascade synthesis on PBEBench Hard
85.8AccuracyBoK + All Symbolic
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| BoK + All SymbolicSystem Type=Hybrid (symbolic + LLM fallback), Tokens (M)=71.6, Cost ($)=19.892026.05 | 85.8 | 0.957 | 92.7 | 4.64 | |
| All SymbolicSystem Type=Symbolic solvers, Tokens (M)=0, Cost ($)=7.112026.05 | 84.7 | 0.992 | 98.6 | 4.56 | |
| BoK + CC + QOSystem Type=Hybrid (symbolic + LLM fallback), Tokens (M)=87.6, Cost ($)=18.482026.05 | 83.5 | 0.9531 | 91.9 | 5.48 | |
| CC + QOSystem Type=Symbolic solvers, Tokens (M)=0, Cost ($)=2.852026.05 | 81.2 | 0.9905 | 98.3 | 5.35 | |
| BoK + QOSystem Type=Hybrid (symbolic + LLM fallback), Tokens (M)=114.5, Cost ($)=21.262026.05 | 80.7 | 0.9496 | 91.3 | 5.48 | |
| BoK + CCSystem Type=Hybrid (symbolic + LLM fallback), Tokens (M)=130.0, Cost ($)=25.082026.05 | 79.4 | 0.9508 | 91.4 | 7.82 | |
| QO SolverSystem Type=Symbolic solvers, Tokens (M)=0, Cost ($)=0.852026.05 | 74.7 | 0.9836 | 96.8 | 5.26 | |
| CC SolverSystem Type=Symbolic solvers, Tokens (M)=0, Cost ($)=2.002026.05 | 69.7 | 0.9873 | 97.2 | 8.06 | |
| BoKSystem Type=LLM-only methods, Tokens (M)=332.1, Cost ($)=57.832026.05 | 68.4 | 0.9428 | 89.9 | 5.14 | |
| Qwen2.5-32B-A14B (OpenHands)System Type=LLM-only methods, Tokens (M)=1046.8, Cost ($)=∼2242026.05 | 24.9 | 0.8411 | 73.2 | 1.36 |