P&ID Question Answering on VectorRAG Conceptual Graph
82AccuracyGPT-5-mini
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GPT-5-miniModel Provider=OpenAI2026.03 | 82 | 0.002 | — | |
| GPT-5Model Provider=OpenAI2026.03 | 79 | 0.017 | — | |
| GPT-OSS:20BModel Provider=Ollama, Hardware Configuration=Mac mini M4 with 24GB RAM2026.03 | 72 | — | 38 | |
| Claude-3-5-HaikuModel Provider=Anthropic2026.03 | 71 | 0.003 | — | |
| Claude-3-7-SonnetModel Provider=Anthropic2026.03 | 71 | 0.015 | — | |
| Claude-Sonnet-4Model Provider=Anthropic2026.03 | 71 | 0.014 | — | |
| Claude-Opus-4-1Model Provider=Anthropic2026.03 | 69 | 0.068 | — | |
| Qwen3:8BModel Provider=Ollama, Hardware Configuration=Mac mini M4 with 24GB RAM2026.03 | 67 | — | 51 | |
| GPT-4o-miniModel Provider=OpenAI2026.03 | 64 | 0.0002 | — | |
| GPT-4oModel Provider=OpenAI2026.03 | 64 | 0.003 | — | |
| Qwen3:4BModel Provider=Ollama, Hardware Configuration=Mac mini M4 with 24GB RAM2026.03 | 63 | — | 54 | |
| Qwen3:14BModel Provider=Ollama, Hardware Configuration=Mac mini M4 with 24GB RAM2026.03 | 63 | — | 84 | |
| LLaMA3.1:8BModel Provider=Ollama, Hardware Configuration=Mac mini M4 with 24GB RAM2026.03 | 59 | — | 13 |