Spatial Reasoning on BBH Navigate
98Accuracy@1DisambiguSLM
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DisambiguSLMLLM=GPT, Avg. Cost ($)=0.022026.04 | 98 | 68.6 | |
| SPOLLM=GPT, Avg. Cost ($)=0.152026.04 | 97.2 | 66.9 | |
| Prompt BreederLLM=GPT, Avg. Cost ($)=4.822026.04 | 96.3 | 64.5 | |
| OPROLLM=GPT, Avg. Cost ($)=4.512026.04 | 95.8 | 66.6 | |
| Prompt AgentLLM=GPT, Avg. Cost ($)=2.712026.04 | 95.7 | 65 | |
| DisambiguSLMLLM=DeepSeek, Avg. Cost ($)=0.022026.04 | 95.4 | 66.3 | |
| DisambiguSLMLLM=LLaMA, Avg. Cost ($)=0.022026.04 | 94.8 | 66 | |
| SPOLLM=DeepSeek, Avg. Cost ($)=0.152026.04 | 94.3 | 64.8 | |
| RephraseLLM=GPT, Avg. Cost ($)=-2026.04 | 93.5 | 61.1 | |
| Step-backLLM=GPT, Avg. Cost ($)=-2026.04 | 93.5 | 65 | |
| SPOLLM=LLaMA, Avg. Cost ($)=0.152026.04 | 93.1 | 64 | |
| Prompt BreederLLM=DeepSeek, Avg. Cost ($)=4.822026.04 | 93 | 62.2 | |
| APELLM=GPT, Avg. Cost ($)=9.072026.04 | 92.5 | 64.8 | |
| OPROLLM=DeepSeek, Avg. Cost ($)=4.512026.04 | 92.4 | 63.9 | |
| Prompt AgentLLM=DeepSeek, Avg. Cost ($)=2.712026.04 | 92.1 | 62.7 | |
| Prompt BreederLLM=LLaMA, Avg. Cost ($)=4.822026.04 | 92 | 61.4 | |
| OPROLLM=LLaMA, Avg. Cost ($)=4.512026.04 | 91.5 | 63.1 | |
| Naïve promptingLLM=GPT, Avg. Cost ($)=-2026.04 | 91.3 | 61.6 | |
| TextGradLLM=GPT, Avg. Cost ($)=13.142026.04 | 91.3 | 63.9 | |
| Step-backLLM=DeepSeek, Avg. Cost ($)=-2026.04 | 91.2 | 62.3 | |
| Prompt AgentLLM=LLaMA, Avg. Cost ($)=2.712026.04 | 91.1 | 62 | |
| RephraseLLM=DeepSeek, Avg. Cost ($)=-2026.04 | 90.8 | 58.8 | |
| Step-backLLM=LLaMA, Avg. Cost ($)=-2026.04 | 90.2 | 61.4 | |
| APELLM=DeepSeek, Avg. Cost ($)=9.072026.04 | 90.1 | 62.4 | |
| CoTLLM=GPT, Avg. Cost ($)=-2026.04 | 89.7 | 63.8 | |
| RephraseLLM=LLaMA, Avg. Cost ($)=-2026.04 | 89.6 | 57.93 | |
| TextGradLLM=DeepSeek, Avg. Cost ($)=13.142026.04 | 89.4 | 61.6 | |
| APELLM=LLaMA, Avg. Cost ($)=9.072026.04 | 89.3 | 61.7 | |
| Naïve promptingLLM=DeepSeek, Avg. Cost ($)=-2026.04 | 88.9 | 59.2 | |
| TextGradLLM=LLaMA, Avg. Cost ($)=13.142026.04 | 88.7 | 60.9 | |
| CoTLLM=DeepSeek, Avg. Cost ($)=-2026.04 | 87.5 | 61.2 | |
| Naïve promptingLLM=LLaMA, Avg. Cost ($)=-2026.04 | 87.1 | 58.2 | |
| CoTLLM=LLaMA, Avg. Cost ($)=-2026.04 | 86.3 | 60.4 |