Question Answering on StrategyQA (EM)
82Exact Match (EM)PTR
Evaluation results
| Method | Links | |
|---|---|---|
| PTRModel=GPT-5.42026.04 | 82 | |
| PTRModel=Claude Sonnet 4.62026.04 | 80 | |
| ReActModel=Claude Haiku 4.52026.04 | 78 | |
| ReActModel=GPT-5.42026.04 | 78 | |
| PTRModel=Claude Haiku 4.52026.04 | 72 | |
| ReActModel=Claude Sonnet 4.62026.04 | 72 | |
| DRAGBase LLM=Llama-3.1-8B-Instruct, Retriever=E5-base-v2, Top-k=3 paragraphs, max debate interactions (r)=3, agents=32026.03 | 69.2 | |
| Naive RAGBase LLM=Llama-3.1-8B-Instruct, Retriever=E5-base-v2, Top-k=3 paragraphs2026.03 | 62.6 | |
| Iter-RetGenBase LLM=Llama-3.1-8B-Instruct, Retriever=E5-base-v2, Top-k=3 paragraphs2026.03 | 62 | |
| VOTE-RAGBase LLM=Llama-3.1-8B-Instruct, Retriever=E5-base-v2, Top-k=3 paragraphs, parallel agents (N)=32026.03 | 61.6 | |
| PTRModel=GPT-4o-Mini2026.04 | 58 | |
| IRCoTBase LLM=Llama-3.1-8B-Instruct, Retriever=E5-base-v2, Top-k=3 paragraphs2026.03 | 53.6 | |
| FLAREBase LLM=Llama-3.1-8B-Instruct, Retriever=E5-base-v2, Top-k=3 paragraphs2026.03 | 42.8 | |
| ReActModel=GPT-4o-Mini2026.04 | 36 | |
| Self-RAGBase LLM=Llama-3.1-8B-Instruct, Retriever=E5-base-v2, Top-k=3 paragraphs2026.03 | 0.4 | |
| SuReBase LLM=Llama-3.1-8B-Instruct, Retriever=E5-base-v2, Top-k=3 paragraphs2026.03 | 0 |