Question Answering on Qasper
67.3RecallPPA-Plan
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| PPA-PlanLLM Model=Llama-3.1-8B2026.01 | 67.3 | — | 75.6 | |
| PPA-PlanLLM Model=GPT-4o mini2026.01 | 67.1 | — | 61.8 | |
| PPA-PlanLLM Model=Qwen-2.5-14B2026.01 | 67.1 | — | 56.4 | |
| PEARLLLM Model=GPT-4o mini2026.01 | 66.5 | — | 51.1 | |
| PEARLLLM Model=Llama-3.1-8B2026.01 | 66 | — | 74.8 | |
| GQALLM Model=GPT-4o mini2026.01 | 65.8 | — | 37.5 | |
| CoTLLM Model=GPT-4o mini2026.01 | 64.9 | — | 46.2 | |
| PEARLLLM Model=Qwen-2.5-14B2026.01 | 63.6 | — | 46.1 | |
| CoTLLM Model=Qwen-2.5-14B2026.01 | 63.4 | — | 36.6 | |
| PSLLM Model=Qwen-2.5-14B2026.01 | 62.8 | — | 39 | |
| GQALLM Model=Qwen-2.5-14B2026.01 | 62.5 | — | 34.9 | |
| PSLLM Model=GPT-4o mini2026.01 | 62.4 | — | 40.5 | |
| CoTLLM Model=Llama-3.1-8B2026.01 | 61.3 | — | 36.6 | |
| GQALLM Model=Llama-3.1-8B2026.01 | 60.5 | — | 31.5 | |
| PSLLM Model=Llama-3.1-8B2026.01 | 58.6 | — | 36.2 | |
| CoABackbone=text-bison, Context Window=8k2024.06 | — | 37.17 | — | |
| CoABackbone=text-unicorn, Context Window=8k2024.06 | — | 38.01 | — | |
| CoABackbone=gemini-ultra, Context Window=8k2024.06 | — | 35.1 | — | |
| RAGBackbone=text-bison, Context Window=8k2024.06 | — | 27.2 | — | |
| RAGBackbone=text-unicorn, Context Window=8k2024.06 | — | 24.44 | — | |
| RAGBackbone=gemini-ultra, Context Window=8k2024.06 | — | 18.7 | — | |
| VanillaBackbone=text-bison, Context Window=8k2024.06 | — | 26.56 | — | |
| VanillaBackbone=text-unicorn, Context Window=8k2024.06 | — | 27.81 | — | |
| VanillaBackbone=gemini-ultra, Context Window=8k2024.06 | — | 20.59 | — | |
| VanillaBackbone=gemini-ultra, Context Window=32k2024.06 | — | 21.71 | — |