Reasoning on SQA
85AccuracyReflexion(3 iters)
Evaluation results
| Method | Links | |
|---|---|---|
| Reflexion(3 iters)Model=Llama3.1-8B, Verification Setting=Oracle2026.03 | 85 | |
| RM-RegenModel=Llama3.1-8B, Verification Setting=Oracle2026.03 | 83.25 | |
| Best-of-N (N=3)Model=Llama3.1-8B, Verification Setting=Oracle2026.03 | 77.5 | |
| RM-RegenModel=GPT-3.5, Verification Setting=Oracle2026.03 | 77.25 | |
| RM-RegenModel=Gemma2-9B, Verification Setting=Oracle2026.03 | 73.5 | |
| RM-RegenModel=Llama3.1-8B2026.03 | 73 | |
| Best-of-N (N=3)Model=GPT-3.5, Verification Setting=Oracle2026.03 | 71.5 | |
| ReflectEvoModel=Llama3.1-8B, Verification Setting=Oracle2026.03 | 71.25 | |
| Reflexion(3 iters)Model=GPT-3.5, Verification Setting=Oracle2026.03 | 71 | |
| RM-RegenModel=GPT-3.52026.03 | 69.75 | |
| ST CoTModel=Llama3.1-8B, iterations=32026.03 | 68 | |
| Reflexion(3 iters)Model=Gemma2-9B, Verification Setting=Oracle2026.03 | 67.5 | |
| RM-RegenModel=Gemma2-9B2026.03 | 66 | |
| ReflectEvoModel=Gemma2-9B, Verification Setting=Oracle2026.03 | 65 | |
| ProCoModel=Gemma2-9B, iterations=32026.03 | 63 | |
| ST CoTModel=GPT-3.5, iterations=32026.03 | 62 | |
| ProCoModel=Llama3.1-8B, iterations=32026.03 | 62 | |
| ProCoModel=GPT-3.5, iterations=32026.03 | 60 | |
| Best-of-N (N=3)Model=Gemma2-9B, Verification Setting=Oracle2026.03 | 53.75 | |
| Self-RefineModel=Llama3.1-8B, iterations=32026.03 | 53.75 | |
| ST CoTModel=Gemma2-9B, iterations=32026.03 | 47.25 | |
| Self-RefineModel=GPT-3.5, iterations=32026.03 | 41.75 | |
| Self-RefineModel=Gemma2-9B, iterations=32026.03 | 28 |