Knowledge Graph Retrieval on STARK Amazon
70.7H@1GRASP
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| GRASPCategory=Hybrid, Setting=Sup2026.05 | 70.7 | 90.2 | 79.1 | 69.1 | |
| KARCategory=Hybrid, Setting=ZS2026.05 | 61.7 | 72.8 | 66.3 | 40.6 | |
| w/o RR f.t.Category=Hybrid, Setting=ZS2026.05 | 59.8 | 81.1 | 69.7 | 67.1 | |
| AvaTaRCategory=Hybrid, Setting=Sup2026.05 | 58 | 76.5 | 65.9 | — | |
| AF-RetrieverCategory=Hybrid, Setting=ZS2026.05 | 58 | 69.1 | 63.5 | — | |
| w/o RRCategory=Hybrid, Setting=ZS2026.05 | 53.1 | 72.6 | 61.9 | 61 | |
| GraphFlowreranking=true2026.05 | 47.9 | 65 | 55.5 | 36.2 | |
| mFAR*Category=Hybrid, Setting=Sup2026.05 | 47.6 | 68.3 | 57.4 | 57.1 | |
| multi-ada-002Category=Textual, Setting=ZS2026.05 | 46.9 | 72.8 | 58.7 | 40.2 | |
| Qwen3/SapBERTCategory=Textual, Setting=ZS2026.05 | 43.9 | 70.7 | 55.2 | 56 | |
| w/o RR & FSCategory=Hybrid, Setting=ZS2026.05 | 41.7 | 58.3 | 50.3 | 46.9 | |
| SeedERtext encoder=Qwen3-4B-Embedding2026.05 | 40.6 | 65.3 | 51.6 | 52.7 | |
| ada-002Category=Textual, Setting=ZS2026.05 | 39.5 | 64.2 | 52.7 | 35.5 | |
| ToG + GPT-4oreranking=true2026.05 | 27.6 | 51.7 | 39.1 | 25.8 | |
| BM25Category=Textual, Setting=ZS2026.05 | 27.2 | 51.9 | 18.8 | 29.2 | |
| QAGNNCategory=Struct., Setting=ZS2026.05 | 22.2 | 49.4 | 31.3 | 21.5 | |
| PRMreranking=true2026.05 | 21.3 | 42.5 | 32 | 35.7 | |
| ToG + GPT-4oreranking=false2026.05 | 20.7 | 41.4 | 30.9 | 25.8 | |
| PRMreranking=false2026.05 | 20.1 | 26.3 | 28.2 | 35.7 | |
| GraphFlowreranking=false2026.05 | 19.6 | 44.2 | 31.7 | 36.2 | |
| DPRCategory=Textual, Setting=ZS2026.05 | 16.1 | 39.5 | 27.2 | 15.2 | |
| SFTreranking=true2026.05 | 12.2 | 30.6 | 21.5 | 25.2 | |
| SFTreranking=false2026.05 | 8.2 | 15.3 | 13.5 | 25.2 | |
| ToG + LLaMA3reranking=false2026.05 | 4.2 | 6.2 | 5.3 | 2.6 | |
| ToG + LLaMA3reranking=true2026.05 | 4.2 | 6.2 | 5.3 | 2.6 |