P&ID Question Answering on ContextRAG Conceptual Graph
94Response AccuracyGPT-5
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GPT-5Model Provider=OpenAI2026.03 | 94 | 0.027 | — | |
| GPT-5-miniModel Provider=OpenAI2026.03 | 91 | 0.004 | — | |
| Claude-3-7-SonnetModel Provider=Anthropic2026.03 | 88 | 0.032 | — | |
| Claude-Sonnet-4Model Provider=Anthropic2026.03 | 87 | 0.033 | — | |
| Claude-Opus-4-1Model Provider=Anthropic2026.03 | 85 | 0.162 | — | |
| GPT-4oModel Provider=OpenAI2026.03 | 83 | 0.027 | — | |
| Claude-3-5-HaikuModel Provider=Anthropic2026.03 | 82 | 0.008 | — | |
| GPT-4o-miniModel Provider=OpenAI2026.03 | 78 | 0.001 | — | |
| Qwen3:14BModel Provider=Ollama, Hardware Configuration=Mac mini M4 with 24GB RAM2026.03 | 43 | — | 284 | |
| Qwen3:4BModel Provider=Ollama, Hardware Configuration=Mac mini M4 with 24GB RAM2026.03 | 37 | — | 275 | |
| Qwen3:8BModel Provider=Ollama, Hardware Configuration=Mac mini M4 with 24GB RAM2026.03 | 35 | — | 173 | |
| GPT-OSS:20BModel Provider=Ollama, Hardware Configuration=Mac mini M4 with 24GB RAM2026.03 | 30 | — | 380 | |
| LLaMA3.1:8BModel Provider=Ollama, Hardware Configuration=Mac mini M4 with 24GB RAM2026.03 | 28 | — | 50 |