List your company
ResearchFormerly WizWand

String transformation on PBEBench Lite

93.9AccuracyBoK + CC
61.24469.72278.286.678May 6, 2026
Updated 6mo ago

Evaluation results

MethodLinks
BoK + CCCategory=Hybrid, LLM Strategy=BoK, Symbolic Solver=CC2026.0593.90.98197.82.9445.610.22
BoK + All SymbolicCategory=Hybrid, LLM Strategy=BoK, Symbolic Solver=All Symbolic2026.0593.90.98197.82.1943.514.94
BoK (gpt-oss-120b)Category=LLM-only, Strategy=BoK, Backbone=gpt-oss-120b2026.0593.80.980897.82.196812.2
BoK + QOCategory=Hybrid, LLM Strategy=BoK, Symbolic Solver=QO2026.0593.80.980897.82.9450.79.97
DF + All SymbolicCategory=Hybrid, LLM Strategy=DF, Symbolic Solver=All Symbolic2026.0593.20.981797.62.1878.718.8
DF + CCCategory=Hybrid, LLM Strategy=DF, Symbolic Solver=CC2026.0593.10.981597.6380.513.97
DF + QOCategory=Hybrid, LLM Strategy=DF, Symbolic Solver=QO2026.0592.90.98197.52.8990.214.35
DF-32 (gpt-oss-120b)Category=LLM-only, Strategy=DF-32, Backbone=gpt-oss-120b2026.0592.40.979697.32.11111.116.74
All SymbolicCategory=Symbolic solver, Method=6 QO solvers + 1 CC solver2026.0591.30.977296.61.8807.11
Qwen3.6-35B-A3B (OpenHands)Category=LLM-only, Framework=OpenHands, Backbone=Qwen3.6-35B-A3B2026.0587.20.954494.91.47395.385.38
CC + QO (union)Category=Symbolic solver, Method=Union of CC and QO2026.0584.60.960794.92.1602.85
CCCategory=Symbolic solver, Induction Model=CC (claude-sonnet-4-6)2026.0580.40.943893.7302
GPT-5 (single)Category=LLM-only, Strategy=single-attempt, Backbone=GPT-52026.0572.4—76.5———
QOCategory=Symbolic solver, Induction Model=QO (Qwen3.6-35B-A3B), Ablation=100 examples + CoT2026.0565.70.902287.43.0100.85
gpt-oss-120b (single)Category=LLM-only, Strategy=single-attempt, Backbone=gpt-oss-120b2026.0562.5—69.9———