Logical Reasoning on ZebraLogic (test)
92.2Grid AccuracyIn-place
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| In-placeBackbone=Qwen, Number of turns=9, Feedback=top-42025.10 | 92.2 | — | — | — | |
| In-placeBackbone=Gemma, Number of turns=9, Feedback=top-42025.10 | 90.6 | — | — | — | |
| In-placeBackbone=Qwen, Number of turns=8, Feedback=top-42025.10 | 89.5 | — | — | — | |
| In-placeBackbone=Gemma, Number of turns=8, Feedback=top-42025.10 | 86.8 | — | — | — | |
| In-placeBackbone=Qwen, Number of turns=7, Feedback=top-42025.10 | 85 | — | — | — | |
| In-placeBackbone=Gemma, Number of turns=7, Feedback=top-42025.10 | 81.6 | — | — | — | |
| In-placeBackbone=Llama, Number of turns=9, Feedback=top-42025.10 | 79 | — | — | — | |
| In-placeBackbone=Qwen, Number of turns=6, Feedback=top-42025.10 | 77.9 | — | — | — | |
| Multi-turnBackbone=Gemma, Number of turns=9, Feedback=top-42025.10 | 77.1 | — | — | — | |
| In-placeBackbone=Llama, Number of turns=8, Feedback=top-42025.10 | 73.8 | — | — | — | |
| Multi-turnBackbone=Gemma, Number of turns=8, Feedback=top-42025.10 | 72.8 | — | — | — | |
| In-placeBackbone=Gemma, Number of turns=6, Feedback=top-42025.10 | 71.2 | — | — | — | |
| Multi-turnBackbone=Qwen, Number of turns=9, Feedback=top-42025.10 | 69 | — | — | — | |
| MemoryBackbone=Llama, Number of turns=9, Feedback=top-42025.10 | 68.2 | — | — | — | |
| In-placeBackbone=Llama, Number of turns=7, Feedback=top-42025.10 | 67.3 | — | — | — | |
| In-placeBackbone=Qwen, Number of turns=5, Feedback=top-42025.10 | 67.2 | — | — | — | |
| MemoryBackbone=Qwen, Number of turns=9, Feedback=top-42025.10 | 66.5 | — | — | — | |
| Multi-turnBackbone=Llama, Number of turns=9, Feedback=top-42025.10 | 65.2 | — | — | — | |
| Multi-turnBackbone=Gemma, Number of turns=7, Feedback=top-42025.10 | 65 | — | — | — | |
| Multi-turnBackbone=Qwen, Number of turns=8, Feedback=top-42025.10 | 64.5 | — | — | — | |
| MemoryBackbone=Llama, Number of turns=8, Feedback=top-42025.10 | 62.5 | — | — | — | |
| Multi-turnBackbone=Llama, Number of turns=8, Feedback=top-42025.10 | 61.2 | — | — | — | |
| MemoryBackbone=Qwen, Number of turns=8, Feedback=top-42025.10 | 60 | — | — | — | |
| In-placeBackbone=Llama, Number of turns=6, Feedback=top-42025.10 | 59 | — | — | — | |
| In-placeBackbone=Gemma, Number of turns=5, Feedback=top-42025.10 | 58.2 | — | — | — | |
| Multi-turnBackbone=Qwen, Number of turns=7, Feedback=top-42025.10 | 56.6 | — | — | — | |
| Multi-turnBackbone=Gemma, Number of turns=6, Feedback=top-42025.10 | 55.6 | — | — | — | |
| Multi-turnBackbone=Llama, Number of turns=7, Feedback=top-42025.10 | 55.1 | — | — | — | |
| MemoryBackbone=Llama, Number of turns=7, Feedback=top-42025.10 | 54.3 | — | — | — | |
| In-placeBackbone=Qwen, Number of turns=4, Feedback=top-42025.10 | 53.4 | — | — | — | |
| MemoryBackbone=Qwen, Number of turns=7, Feedback=top-42025.10 | 52 | — | — | — | |
| In-placeBackbone=Llama, Number of turns=5, Feedback=top-42025.10 | 47.8 | — | — | — | |
| Multi-turnBackbone=Llama, Number of turns=6, Feedback=top-42025.10 | 47 | — | — | — | |
| Multi-turnBackbone=Qwen, Number of turns=6, Feedback=top-42025.10 | 46.8 | — | — | — | |
| MemoryBackbone=Llama, Number of turns=6, Feedback=top-42025.10 | 45.7 | — | — | — | |
| In-placeBackbone=Gemma, Number of turns=4, Feedback=top-42025.10 | 44.1 | — | — | — | |
| MemoryBackbone=Qwen, Number of turns=6, Feedback=top-42025.10 | 43.4 | — | — | — | |
| Multi-turnBackbone=Gemma, Number of turns=5, Feedback=top-42025.10 | 42.1 | — | — | — | |
| MemoryBackbone=Llama, Number of turns=5, Feedback=top-42025.10 | 36.9 | — | — | — | |
| In-placeBackbone=Llama, Number of turns=4, Feedback=top-42025.10 | 35.9 | — | — | — | |
| Multi-turnBackbone=Llama, Number of turns=5, Feedback=top-42025.10 | 35.8 | — | — | — | |
| Multi-turnBackbone=Qwen, Number of turns=5, Feedback=top-42025.10 | 35.6 | — | — | — | |
| MemoryBackbone=Qwen, Number of turns=5, Feedback=top-42025.10 | 34.8 | — | — | — | |
| In-placeBackbone=Qwen, Number of turns=3, Feedback=top-42025.10 | 34.1 | — | — | — | |
| Multi-turnBackbone=Gemma, Number of turns=4, Feedback=top-42025.10 | 27.1 | — | — | — | |
| In-placeBackbone=Gemma, Number of turns=3, Feedback=top-42025.10 | 26.5 | — | — | — | |
| MemoryBackbone=Llama, Number of turns=4, Feedback=top-42025.10 | 26.4 | — | — | — | |
| MemoryBackbone=Qwen, Number of turns=4, Feedback=top-42025.10 | 25.1 | — | — | — | |
| Multi-turnBackbone=Qwen, Number of turns=4, Feedback=top-42025.10 | 24.9 | — | — | — | |
| MemoryBackbone=Gemma, Number of turns=9, Feedback=top-42025.10 | 23.4 | — | — | — | |
| Multi-turnBackbone=Llama, Number of turns=4, Feedback=top-42025.10 | 22.5 | — | — | — | |
| In-placeBackbone=Llama, Number of turns=3, Feedback=top-42025.10 | 22.3 | — | — | — | |
| MemoryBackbone=Gemma, Number of turns=8, Feedback=top-42025.10 | 19.9 | — | — | — | |
| MemoryBackbone=Gemma, Number of turns=7, Feedback=top-42025.10 | 16.7 | — | — | — | |
| MemoryBackbone=Qwen, Number of turns=3, Feedback=top-42025.10 | 16.6 | — | — | — | |
| MemoryBackbone=Llama, Number of turns=3, Feedback=top-42025.10 | 16 | — | — | — | |
| Multi-turnBackbone=Qwen, Number of turns=3, Feedback=top-42025.10 | 15 | — | — | — | |
| Multi-turnBackbone=Gemma, Number of turns=3, Feedback=top-42025.10 | 14.8 | — | — | — | |
| In-placeBackbone=Qwen, Number of turns=2, Feedback=top-42025.10 | 14.4 | — | — | — | |
| MemoryBackbone=Gemma, Number of turns=6, Feedback=top-42025.10 | 13.5 | — | — | — | |
| Multi-turnBackbone=Llama, Number of turns=3, Feedback=top-42025.10 | 12.6 | — | — | — | |
| In-placeBackbone=Gemma, Number of turns=2, Feedback=top-42025.10 | 11.5 | — | — | — | |
| MemoryBackbone=Gemma, Number of turns=5, Feedback=top-42025.10 | 10.5 | — | — | — | |
| In-placeBackbone=Llama, Number of turns=2, Feedback=top-42025.10 | 10.5 | — | — | — | |
| Multi-turnBackbone=Gemma, Number of turns=2, Feedback=top-42025.10 | 7.8 | — | — | — | |
| MemoryBackbone=Gemma, Number of turns=4, Feedback=top-42025.10 | 7.8 | — | — | — | |
| MemoryBackbone=Llama, Number of turns=2, Feedback=top-42025.10 | 7.6 | — | — | — | |
| MemoryBackbone=Qwen, Number of turns=2, Feedback=top-42025.10 | 7.3 | — | — | — | |
| Multi-turnBackbone=Qwen, Number of turns=2, Feedback=top-42025.10 | 7.1 | — | — | — | |
| Multi-turnBackbone=Llama, Number of turns=2, Feedback=top-42025.10 | 5.6 | — | — | — | |
| MemoryBackbone=Gemma, Number of turns=3, Feedback=top-42025.10 | 5.2 | — | — | — | |
| In-placeBackbone=Qwen, Number of turns=1, Feedback=top-42025.10 | 3.6 | — | — | — | |
| In-placeBackbone=Llama, Number of turns=1, Feedback=top-42025.10 | 3.4 | — | — | — | |
| In-placeBackbone=Gemma, Number of turns=1, Feedback=top-42025.10 | 3.1 | — | — | — | |
| Multi-turnBackbone=Gemma, Number of turns=1, Feedback=top-42025.10 | 2.8 | — | — | — | |
| MemoryBackbone=Gemma, Number of turns=2, Feedback=top-42025.10 | 2.8 | — | — | — | |
| MemoryBackbone=Qwen, Number of turns=1, Feedback=top-42025.10 | 2.5 | — | — | — | |
| MemoryBackbone=Llama, Number of turns=1, Feedback=top-42025.10 | 2.3 | — | — | — | |
| Multi-turnBackbone=Qwen, Number of turns=1, Feedback=top-42025.10 | 2.2 | — | — | — | |
| Multi-turnBackbone=Llama, Number of turns=1, Feedback=top-42025.10 | 1.7 | — | — | — | |
| MemoryBackbone=Gemma, Number of turns=1, Feedback=top-42025.10 | 0.7 | — | — | — | |
| Multi-turnBackbone=Llama, Number of turns=0, Feedback=top-42025.10 | 0.6 | — | — | — | |
| MemoryBackbone=Llama, Number of turns=0, Feedback=top-42025.10 | 0.6 | — | — | — | |
| In-placeBackbone=Llama, Number of turns=0, Feedback=top-42025.10 | 0.6 | — | — | — | |
| MemoryBackbone=Qwen, Number of turns=0, Feedback=top-42025.10 | 0.4 | — | — | — | |
| In-placeBackbone=Qwen, Number of turns=0, Feedback=top-42025.10 | 0.4 | — | — | — | |
| Multi-turnBackbone=Qwen, Number of turns=0, Feedback=top-42025.10 | 0.3 | — | — | — | |
| Multi-turnBackbone=Gemma, Number of turns=0, Feedback=top-42025.10 | 0 | — | — | — | |
| MemoryBackbone=Gemma, Number of turns=0, Feedback=top-42025.10 | 0 | — | — | — | |
| In-placeBackbone=Gemma, Number of turns=0, Feedback=top-42025.10 | 0 | — | — | — | |
| G-PACScore type=Logits-based score, LLM Backbone=Llama-based, Alpha=0.05, Epsilon=0.22026.01 | — | 11.43 | 0 | 13.23 | |
| G-PACScore type=Verbalized score, LLM Backbone=Llama-based, Alpha=0.05, Epsilon=0.22026.01 | — | 11.13 | 0 | -9.52 | |
| G-PACScore type=Router-based score, LLM Backbone=Llama-based, Alpha=0.05, Epsilon=0.22026.01 | — | 11.13 | 0 | 25.98 | |
| PACScore type=Logits-based score, LLM Backbone=Llama-based, Alpha=0.05, Epsilon=0.22026.01 | — | 17.33 | 16.2 | 23.72 | |
| PACScore type=Verbalized score, LLM Backbone=Llama-based, Alpha=0.05, Epsilon=0.22026.01 | — | 17.05 | 15.37 | -30.68 | |
| PACScore type=Router-based score, LLM Backbone=Llama-based, Alpha=0.05, Epsilon=0.22026.01 | — | 17.2 | 15.55 | 25.22 |