String transformation on PBEBench Lite
93.9AccuracyBoK + CC
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| BoK + CCCategory=Hybrid, LLM Strategy=BoK, Symbolic Solver=CC2026.05 | 93.9 | 0.981 | 97.8 | 2.94 | 45.6 | 10.22 | |
| BoK + All SymbolicCategory=Hybrid, LLM Strategy=BoK, Symbolic Solver=All Symbolic2026.05 | 93.9 | 0.981 | 97.8 | 2.19 | 43.5 | 14.94 | |
| BoK (gpt-oss-120b)Category=LLM-only, Strategy=BoK, Backbone=gpt-oss-120b2026.05 | 93.8 | 0.9808 | 97.8 | 2.19 | 68 | 12.2 | |
| BoK + QOCategory=Hybrid, LLM Strategy=BoK, Symbolic Solver=QO2026.05 | 93.8 | 0.9808 | 97.8 | 2.94 | 50.7 | 9.97 | |
| DF + All SymbolicCategory=Hybrid, LLM Strategy=DF, Symbolic Solver=All Symbolic2026.05 | 93.2 | 0.9817 | 97.6 | 2.18 | 78.7 | 18.8 | |
| DF + CCCategory=Hybrid, LLM Strategy=DF, Symbolic Solver=CC2026.05 | 93.1 | 0.9815 | 97.6 | 3 | 80.5 | 13.97 | |
| DF + QOCategory=Hybrid, LLM Strategy=DF, Symbolic Solver=QO2026.05 | 92.9 | 0.981 | 97.5 | 2.89 | 90.2 | 14.35 | |
| DF-32 (gpt-oss-120b)Category=LLM-only, Strategy=DF-32, Backbone=gpt-oss-120b2026.05 | 92.4 | 0.9796 | 97.3 | 2.11 | 111.1 | 16.74 | |
| All SymbolicCategory=Symbolic solver, Method=6 QO solvers + 1 CC solver2026.05 | 91.3 | 0.9772 | 96.6 | 1.88 | 0 | 7.11 | |
| Qwen3.6-35B-A3B (OpenHands)Category=LLM-only, Framework=OpenHands, Backbone=Qwen3.6-35B-A3B2026.05 | 87.2 | 0.9544 | 94.9 | 1.47 | 395.3 | 85.38 | |
| CC + QO (union)Category=Symbolic solver, Method=Union of CC and QO2026.05 | 84.6 | 0.9607 | 94.9 | 2.16 | 0 | 2.85 | |
| CCCategory=Symbolic solver, Induction Model=CC (claude-sonnet-4-6)2026.05 | 80.4 | 0.9438 | 93.7 | 3 | 0 | 2 | |
| GPT-5 (single)Category=LLM-only, Strategy=single-attempt, Backbone=GPT-52026.05 | 72.4 | — | 76.5 | — | — | — | |
| QOCategory=Symbolic solver, Induction Model=QO (Qwen3.6-35B-A3B), Ablation=100 examples + CoT2026.05 | 65.7 | 0.9022 | 87.4 | 3.01 | 0 | 0.85 | |
| gpt-oss-120b (single)Category=LLM-only, Strategy=single-attempt, Backbone=gpt-oss-120b2026.05 | 62.5 | — | 69.9 | — | — | — |