Reasoning on GPQA (Acc@1, Avg. Perf.)
44Accuracy@1 (GPQA)DisambiguSLM
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DisambiguSLMLLM=GPT, Avg. Cost ($)=0.022026.04 | 44 | 68.6 | |
| SPOLLM=GPT, Avg. Cost ($)=0.152026.04 | 43.6 | 66.9 | |
| OPROLLM=GPT, Avg. Cost ($)=4.512026.04 | 43.3 | 66.6 | |
| Step-backLLM=GPT, Avg. Cost ($)=-2026.04 | 42.4 | 65 | |
| CoTLLM=GPT, Avg. Cost ($)=-2026.04 | 41.6 | 63.8 | |
| DisambiguSLMLLM=DeepSeek, Avg. Cost ($)=0.022026.04 | 41.6 | 66.3 | |
| Prompt AgentLLM=GPT, Avg. Cost ($)=2.712026.04 | 41.3 | 65 | |
| APELLM=GPT, Avg. Cost ($)=9.072026.04 | 41.1 | 64.8 | |
| DisambiguSLMLLM=LLaMA, Avg. Cost ($)=0.022026.04 | 41.1 | 66 | |
| Prompt BreederLLM=GPT, Avg. Cost ($)=4.822026.04 | 40.9 | 64.5 | |
| SPOLLM=DeepSeek, Avg. Cost ($)=0.152026.04 | 40.5 | 64.8 | |
| RephraseLLM=GPT, Avg. Cost ($)=-2026.04 | 40.2 | 61.1 | |
| TextGradLLM=GPT, Avg. Cost ($)=13.142026.04 | 40.2 | 63.9 | |
| OPROLLM=DeepSeek, Avg. Cost ($)=4.512026.04 | 40.1 | 63.9 | |
| SPOLLM=LLaMA, Avg. Cost ($)=0.152026.04 | 40 | 64 | |
| Step-backLLM=DeepSeek, Avg. Cost ($)=-2026.04 | 39.8 | 62.3 | |
| OPROLLM=LLaMA, Avg. Cost ($)=4.512026.04 | 39.6 | 63.1 | |
| Step-backLLM=LLaMA, Avg. Cost ($)=-2026.04 | 39.1 | 61.4 | |
| Naïve promptingLLM=GPT, Avg. Cost ($)=-2026.04 | 38.9 | 61.6 | |
| CoTLLM=DeepSeek, Avg. Cost ($)=-2026.04 | 38.7 | 61.2 | |
| Prompt AgentLLM=DeepSeek, Avg. Cost ($)=2.712026.04 | 38.6 | 62.7 | |
| APELLM=DeepSeek, Avg. Cost ($)=9.072026.04 | 38.5 | 62.4 | |
| CoTLLM=LLaMA, Avg. Cost ($)=-2026.04 | 38.2 | 60.4 | |
| Prompt AgentLLM=LLaMA, Avg. Cost ($)=2.712026.04 | 38.2 | 62 | |
| Prompt BreederLLM=DeepSeek, Avg. Cost ($)=4.822026.04 | 38.1 | 62.2 | |
| APELLM=LLaMA, Avg. Cost ($)=9.072026.04 | 38 | 61.7 | |
| RephraseLLM=DeepSeek, Avg. Cost ($)=-2026.04 | 37.9 | 58.8 | |
| Prompt BreederLLM=LLaMA, Avg. Cost ($)=4.822026.04 | 37.6 | 61.4 | |
| TextGradLLM=DeepSeek, Avg. Cost ($)=13.142026.04 | 37.5 | 61.6 | |
| RephraseLLM=LLaMA, Avg. Cost ($)=-2026.04 | 37.1 | 57.93 | |
| TextGradLLM=LLaMA, Avg. Cost ($)=13.142026.04 | 36.9 | 60.9 | |
| Naïve promptingLLM=DeepSeek, Avg. Cost ($)=-2026.04 | 36.4 | 59.2 | |
| Naïve promptingLLM=LLaMA, Avg. Cost ($)=-2026.04 | 35.8 | 58.2 |