Biomedical Evidence Synthesis on COCHRANEFOREST (test)
67.3F1 ScoreURCA
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| URCALLM=Mistral-Large (2407), 128B2025.05 | 67.3 | 65.2 | 69.5 | 67 | |
| URCALLM=LlaMa 3.1, 70B2025.05 | 66.1 | 64 | 68.4 | 64.6 | |
| URCALLM=GPT-4 (0613)2025.05 | 65.7 | 63 | 68.7 | 64.1 | |
| GraphRAGLLM=LlaMa 3.1, 70B2025.05 | 65.6 | 64.8 | 66.3 | 63.4 | |
| GraphRAGLLM=Mistral-Large (2407), 128B2025.05 | 64.9 | 63.8 | 66 | 66.3 | |
| GraphRAGLLM=GPT-4 (0613)2025.05 | 63.8 | 61.6 | 66.1 | 63.7 | |
| RAG + Uniform RetrievalLLM=LlaMa 3.1, 70B2025.05 | 63.4 | 61.5 | 65.5 | 64.2 | |
| RAG + Uniform RetrievalLLM=Mistral-Large (2407), 128B2025.05 | 63.3 | 60.9 | 65.8 | 63.2 | |
| AbstractsLLM=Mistral-Large (2407), 128B2025.05 | 62.6 | 60.3 | 65 | 62.6 | |
| InstructRAGLLM=Mistral-Large (2407), 128B2025.05 | 62.6 | 61.5 | 63.6 | 64.4 | |
| URCALLM=GPT-3.5-Turbo (0613)2025.05 | 62.4 | 60.6 | 64.4 | 63.5 | |
| RAGLLM=LlaMa 3.1, 70B2025.05 | 62.1 | 60.2 | 64.1 | 62.9 | |
| RAG + Uniform RetrievalLLM=GPT-4 (0613)2025.05 | 62 | 59.7 | 64.5 | 61.8 | |
| RAG + Uniform RetrievalLLM=GPT-3.5-Turbo (0613)2025.05 | 61.8 | 60 | 63.8 | 62.8 | |
| RAPTORLLM=Mistral-Large (2407), 128B2025.05 | 61.7 | 59.1 | 64.6 | 61 | |
| RAGLLM=GPT-4 (0613)2025.05 | 61.6 | 59.3 | 64 | 61.6 | |
| InstructRAGLLM=GPT-4 (0613)2025.05 | 61.6 | 61.4 | 61.9 | 63.3 | |
| AbstractsLLM=GPT-4 (0613)2025.05 | 61 | 60.2 | 61.9 | 62.6 | |
| RAGLLM=Mistral-Large (2407), 128B2025.05 | 60.9 | 60.7 | 61 | 64.3 | |
| InstructRAGLLM=LlaMa 3.1, 70B2025.05 | 60.9 | 59.6 | 62.3 | 62.1 | |
| AbstractsLLM=LlaMa 3.1, 70B2025.05 | 60.7 | 60.4 | 61.1 | 64.1 | |
| RAPTORLLM=LlaMa 3.1, 70B2025.05 | 60.6 | 58.2 | 63.2 | 60 | |
| RAPTORLLM=GPT-4 (0613)2025.05 | 60.1 | 58.2 | 62 | 60.8 | |
| RAGLLM=GPT-3.5-Turbo (0613)2025.05 | 59.1 | 57.6 | 60.6 | 60.7 | |
| InstructRAGLLM=GPT-3.5-Turbo (0613)2025.05 | 57.4 | 57.1 | 57.7 | 61.1 | |
| GraphRAGLLM=GPT-3.5-Turbo (0613)2025.05 | 56.6 | 58.4 | 54.9 | 59.3 | |
| AbstractsLLM=GPT-3.5-Turbo (0613)2025.05 | 56 | 55.2 | 56.9 | 58.8 | |
| RAPTORLLM=GPT-3.5-Turbo (0613)2025.05 | 53.6 | 52 | 55.3 | 54.1 | |
| No RAGLLM=LlaMa 3.1, 70B2025.05 | 49.1 | 46.1 | 52.4 | 45.2 | |
| No RAGLLM=GPT-4 (0613)2025.05 | 47.5 | 55.9 | 41.2 | 59.5 | |
| No RAGLLM=Mistral-Large (2407), 128B2025.05 | 46.1 | 47.4 | 44.8 | 53.1 | |
| No RAGLLM=GPT-3.5-Turbo (0613)2025.05 | 24.1 | 18.9 | 33.2 | 56.4 |