Logical Puzzle Solving on PPBench (golden)
97.8Sudoku Success RatePTRM
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| PTRMStrategy=best-Q@K, Parameters=7M2026.05 | 97.8 | 100 | 88.9 | 85.7 | 80 | 91.2 | 0.001 | 0.001 | |
| LLM ensembleStrategy=Any strategy (direct or agentic)2026.05 | 46.7 | 100 | 44.4 | 0 | 80 | 55.1 | 2.66 | 38.51 | |
| gpt-5.2@xhighStrategy=Any strategy (direct or agentic)2026.05 | 33.3 | 75 | 0 | 0 | 60 | 34.7 | 3.09 | 8.9 | |
| gpt-5.2@xhighStrategy=Direct2026.05 | 20 | 50 | 0 | 0 | 50 | 24.5 | 1.79 | 7.29 | |
| gemini-3.1-proStrategy=Direct2026.05 | 6.7 | 75 | 22.2 | 0 | 30 | 24.5 | 0.4 | 1.62 | |
| gemini-3.1-proStrategy=Any strategy (direct or agentic)2026.05 | 6.7 | 87.5 | 33.3 | 0 | 40 | 30.6 | 10.38 | 33.91 | |
| claude-opus-4-6@thinkingStrategy=Direct2026.05 | 0 | 87.5 | 44.4 | 0 | 60 | 34.7 | 2.91 | 8.4 | |
| claude-opus-4-6@thinkingStrategy=Any strategy (direct or agentic)2026.05 | 0 | 87.5 | 44.4 | 0 | 70 | 36.7 | 4.38 | 11.92 |