Temporal Knowledge Graph Question Answering on CRONQUESTIONS (test)
97.8Hits@1 (Overall)GenTKGQA
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GenTKGQA2024.02 | 97.8 | 96.2 | 99.9 | 96.7 | 99 | 98.3 | 97.1 | 99.9 | 97.4 | 99.4 | — | — | — | — | |
| QC-MHM2024.02 | 97.1 | 94.6 | 99.2 | 96.2 | 96.6 | 99.2 | 98.6 | 99.8 | 99.3 | 98.7 | — | — | — | — | |
| Prog-TQA2024.04 | 93.7 | 89.8 | 98.9 | 91.4 | 98.2 | 97.3 | 96 | 99.3 | 96.8 | 98.2 | — | — | — | — | |
| CTRN2024.02 | 92 | 86.9 | 90 | 92.1 | 91.7 | 98 | 97 | 99.3 | 98.2 | 97.6 | — | — | — | — | |
| TempoQR-HardSupervision=Hard2021.12 | 91.8 | 86.4 | 99 | 92.6 | 90.3 | 97.8 | 96.7 | 99.3 | 98 | 97.4 | — | — | — | — | |
| TempoQR2024.02 | 91.8 | 86.4 | 99 | 92.6 | 90.3 | 97.8 | 97.8 | 99.3 | 98 | 97.4 | — | — | — | — | |
| TempoQR2024.04 | 91.8 | 86.4 | 99 | 92.6 | 90.3 | 97.8 | 96.7 | 99.3 | 98 | 97.4 | — | — | — | — | |
| TempoQR2024.02 | 91.8 | 86.4 | 99 | 92.6 | 90.3 | 97.8 | 96.7 | 99.3 | 98 | 97.4 | — | — | — | — | |
| TKG-ThinkerBackbone=Llama3-8B-Instruct, Training Protocol=SFT+GRPO2026.02 | 91.5 | 87.5 | 96.8 | 89.3 | 95.5 | — | — | — | — | — | — | — | — | — | |
| TKG-ThinkerBackbone=Qwen2.5-7B-Instruct, Training Protocol=SFT+GRPO2026.02 | 89.3 | 84.4 | 95.8 | 86.3 | 94.9 | — | — | — | — | — | — | — | — | — | |
| TKG-ThinkerBackbone=Qwen3-4B-Instruct, Training Protocol=SFT+GRPO2026.02 | 88.9 | 83.7 | 95.3 | 85.7 | 94.8 | — | — | — | — | — | — | — | — | — | |
| TKG-ThinkerBackbone=Qwen3-4B-Instruct, Training Protocol=SFT+PPO2026.02 | 87.7 | 82 | 95.3 | 84.1 | 94.4 | — | — | — | — | — | — | — | — | — | |
| TKG-ThinkerBackbone=Llama3-8B-Instruct, Training Protocol=SFT+PPO2026.02 | 87.3 | 81.7 | 94.9 | 84.4 | 92.8 | — | — | — | — | — | — | — | — | — | |
| TKG-ThinkerBackbone=Qwen2.5-7B-Instruct, Training Protocol=SFT+PPO2026.02 | 86.8 | 81.7 | 93.7 | 83.8 | 92.5 | — | — | — | — | — | — | — | — | — | |
| TempAgentModel Category=LLM-based Methods2026.02 | 84.2 | 64 | 89.5 | 80.5 | 92.1 | — | — | — | — | — | — | — | — | — | |
| TSQA2022.03 | 83.1 | 71.3 | 98.7 | 82.9 | 83.6 | 98 | 96.8 | 99.7 | 98.1 | 97.8 | — | — | — | — | |
| TSQA2024.02 | 83.1 | 71.3 | 98.7 | 82.9 | 83.6 | 98 | 96.8 | 99.7 | 98.1 | 97.8 | — | — | — | — | |
| TSQA2024.02 | 83.1 | 71.3 | 98.7 | 82.9 | 83.6 | 98 | 96.8 | 99.7 | 98.1 | 97.8 | — | — | — | — | |
| ReAct RAGModel Category=LLM-based Methods2026.02 | 80.9 | 60 | 86.3 | 76.8 | 89.5 | — | — | — | — | — | — | — | — | — | |
| TempoQR-SoftSupervision=Soft2021.12 | 79.9 | 65.5 | 99 | 87.6 | 65.3 | 95.7 | 93 | 99.3 | 97.2 | 92.9 | — | — | — | — | |
| TMA2023.02 | 78.4 | 63.2 | 98.7 | 79.2 | 74.3 | 94.3 | 90.4 | 99.5 | 94.7 | 93.6 | — | — | — | — | |
| TMA2024.02 | 78.4 | 63.2 | 98.7 | 79.2 | 74.3 | 94.3 | 90.4 | 99.5 | 94.7 | 93.6 | — | — | — | — | |
| TMA2024.04 | 78.4 | 63.2 | 98.7 | 79.2 | 74.3 | 94.3 | 90.4 | 99.5 | 94.7 | 93.6 | — | — | — | — | |
| TMA2024.02 | 78.4 | 63.2 | 98.7 | 79.2 | 74.3 | 94.3 | 90.4 | 99.5 | 94.7 | 93.6 | — | — | — | — | |
| ChatGPTTKG background knowledge=with2024.02 | 75.4 | 57.9 | 98.7 | 68.9 | 87.3 | 85.2 | 74.6 | 99.2 | 80.8 | 93.3 | — | — | — | — | |
| EntityQR2021.12 | 74.5 | 56.2 | 99 | 83.1 | 58.5 | 94.4 | 90.6 | 99.3 | 96.2 | 91 | — | — | — | — | |
| EntityQR2024.02 | 74.5 | 56.2 | 99 | 83.1 | 58.5 | 94.4 | 90.6 | 99.3 | 96.2 | 91 | — | — | — | — | |
| ARIModel Category=LLM-based Methods2026.02 | 70.7 | 57 | 86 | 66 | 80 | — | — | — | — | — | — | — | — | — | |
| CRONKGQA2021.06 | 64.7 | 39.2 | 98.7 | 69.9 | 54.9 | 88.4 | 80.2 | 99.2 | 89.8 | 85.7 | — | — | — | — | |
| CronKGQA2021.12 | 64.7 | 39.2 | 98.7 | 69.9 | 54.9 | 88.4 | 80.2 | 99.2 | 89.8 | 85.7 | — | — | — | — | |
| CronKGQA2022.03 | 64.7 | 39.2 | 98.7 | 69.9 | 54.9 | 88.4 | 80.2 | 99.2 | 89.8 | 85.7 | — | — | — | — | |
| CronKGQA2023.02 | 64.7 | 39.2 | 98.7 | 69.9 | 54.9 | 88.4 | 80.2 | 99.2 | 89.8 | 85.7 | — | — | — | — | |
| CronKGQA2024.02 | 64.7 | 39.2 | 98.7 | 69.9 | 54.9 | 88.4 | 80.2 | 99.2 | 89.8 | 85.7 | — | — | — | — | |
| CronKGQA2024.04 | 64.7 | 39.2 | 98.7 | 69.9 | 54.9 | 88.4 | 80.2 | 99.2 | 89.8 | 85.7 | — | — | — | — | |
| CronKGQA2024.02 | 64.7 | 39.2 | 98.7 | 69.9 | 54.9 | 88.4 | 80.2 | 99 | 89.8 | 85.7 | — | — | — | — | |
| CronKGQAModel Category=Embedding-based Methods2026.02 | 64.7 | 39.2 | 98.7 | 69.9 | 54.9 | — | — | — | — | — | — | — | — | — | |
| Naive RAGModel Category=LLM-based Methods2026.02 | 63.3 | 28 | 72.6 | 61 | 68.4 | — | — | — | — | — | — | — | — | — | |
| EmbedKGQA2021.06 | 28.8 | 28.6 | 29 | 41.1 | 5.7 | 67.2 | 63.2 | 72.5 | 85 | 34.1 | — | — | — | — | |
| T-EaE-replacevariant=replacement2021.06 | 28.8 | 25.7 | 32.9 | 31.8 | 23.1 | 67.8 | 62.3 | 75.3 | 66.8 | 69.8 | — | — | — | — | |
| EmbedKGQA2021.12 | 28.8 | 28.6 | 29 | 41.1 | 5.7 | 67.2 | 63.2 | 72.5 | 85 | 34.1 | — | — | — | — | |
| EaE2021.12 | 28.8 | 25.7 | 32.9 | 31.8 | 23.1 | 67.8 | 62.3 | 75.3 | 68.8 | 69.8 | — | — | — | — | |
| EmbedKGQA2022.03 | 28.8 | 28.6 | 29 | 41.1 | 5.7 | 67.2 | 63.2 | 72.5 | 85 | 34.1 | — | — | — | — | |
| T-EaE-replace2022.03 | 28.8 | 25.7 | 32.9 | 31.8 | 23.1 | 67.8 | 62.3 | 75.3 | 66.8 | 69.8 | — | — | — | — | |
| EmbedKGQA2023.02 | 28.8 | 28.6 | 29 | 41.1 | 5.7 | 67.2 | 63.2 | 72.5 | 85 | 34.1 | — | — | — | — | |
| T-EaE-replace2023.02 | 28.8 | 25.7 | 32.9 | 31.8 | 23.1 | 67.8 | 62.3 | 75.3 | 66.8 | 69.8 | — | — | — | — | |
| EmbedKGQA2024.02 | 28.8 | 28.6 | 29 | 41.1 | 5.7 | 67.2 | 63.2 | 72.5 | 85 | 34.1 | — | — | — | — | |
| T-EaE-replace2024.02 | 28.8 | 25.7 | 32.9 | 31.8 | 23.1 | 67.8 | 62.3 | 75.3 | 66.8 | 69.8 | — | — | — | — | |
| EmbedKGQA2024.04 | 28.8 | 28.6 | 29 | 41.1 | 5.7 | 67.2 | 63.2 | 72.5 | 85 | 34.1 | — | — | — | — | |
| EmbedKGQA2024.02 | 28.8 | 28.6 | 29 | 41.1 | 5.7 | 67.2 | 63.2 | 72.5 | 85 | 34.1 | — | — | — | — | |
| EaE2024.02 | 28.8 | 25.7 | 32.9 | 31.8 | 23.1 | 67.8 | 62.3 | 75.3 | 66.8 | 69.8 | — | — | — | — | |
| EmbedKGQAModel Category=Embedding-based Methods2026.02 | 28.8 | 28.6 | 29 | 41.1 | 5.7 | — | — | — | — | — | — | — | — | — | |
| T-EaE-addvariant=additive2021.06 | 27.8 | 25.7 | 30.6 | 31.3 | 21.3 | 66.3 | 61.4 | 72.9 | 66.2 | 66.5 | — | — | — | — | |
| T-EaE-add2022.03 | 27.8 | 25.7 | 30.6 | 31.3 | 21.3 | 66.3 | 61.4 | 72.9 | 66.2 | 66.5 | — | — | — | — | |
| T-EaE-add2023.02 | 27.8 | 25.7 | 30.6 | 31.3 | 21.3 | 66.3 | 61.4 | 72.9 | 66.2 | 66.5 | — | — | — | — | |
| T-EaE-add2024.02 | 27.8 | 25.7 | 30.6 | 31.3 | 21.3 | 66.3 | 61.4 | 72.9 | 66.2 | 66.5 | — | — | — | — | |
| T5-3B2024.02 | 25.2 | 24 | 25.1 | 28.3 | 18 | — | — | — | — | — | — | — | — | — | |
| ALBERTModel Category=PLM-based Methods2026.02 | 24.8 | 23.5 | 25.5 | 27.9 | 17.7 | — | — | — | — | — | — | — | — | — | |
| BERT2021.12 | 24.3 | 23.9 | 24.9 | 27.7 | 17.9 | 62 | 59.8 | 64.9 | 62.8 | 60.4 | — | — | — | — | |
| BERT2024.02 | 24.3 | 23.9 | 24.9 | 27.7 | 17.9 | 62 | 59.8 | 64.9 | 62.8 | 60.4 | — | — | — | — | |
| BERT2024.04 | 24.3 | 23.9 | 24.9 | 27.7 | 17.9 | 62 | 59.8 | 64.9 | 62.8 | 60.4 | — | — | — | — | |
| BERTTKG background knowledge=with2024.02 | 24.3 | 23.9 | 24.9 | 27.7 | 17.9 | 62 | 59.8 | 64.9 | 62.8 | 60.4 | — | — | — | — | |
| BERTModel Category=PLM-based Methods2026.02 | 24.3 | 23.9 | 24.9 | 27.7 | 17.9 | — | — | — | — | — | — | — | — | — | |
| KnowBERT2024.02 | 22.6 | 22 | 23.8 | 25.2 | 17.7 | 58.6 | 53.9 | 64.6 | 58.2 | 59.2 | — | — | — | — | |
| RoBERTa2021.12 | 22.5 | 21.7 | 23.7 | 25.1 | 17.7 | 58.5 | 54.2 | 64.4 | 58.3 | 59.1 | — | — | — | — | |
| RoBERTa2024.02 | 22.5 | 21.7 | 23.7 | 25.1 | 17.7 | 58.5 | 54.2 | 64.4 | 58.3 | 59.1 | — | — | — | — | |
| RoBERTaTKG background knowledge=with2024.02 | 22.5 | 21.7 | 23.7 | 25.1 | 17.7 | 58.5 | 54.2 | 64.4 | 58.3 | 59.1 | — | — | — | — | |
| ChatGPTTKG background knowledge=without2024.02 | 15.1 | 14.4 | 16 | 13.4 | 18.2 | 30.8 | 30.8 | 30.7 | 25.7 | 40.2 | — | — | — | — | |
| T5-3Bmodel_size=3B2021.06 | 8.1 | 7.3 | 9.1 | 8.8 | 6.7 | — | — | — | — | — | — | — | — | — | |
| T5-3BModel Size=3B2023.02 | 8.1 | 7.3 | 9.1 | 8.8 | 6.7 | — | — | — | — | — | — | — | — | — | |
| BERT2021.06 | 7.1 | 8.6 | 5.2 | 7.7 | 6 | 21.3 | 20.5 | 22.5 | 19.2 | 25.3 | — | — | — | — | |
| BERT2023.02 | 7.1 | 8.6 | 5.2 | 7.7 | 6 | 21.3 | 20.5 | 22.5 | 19.2 | 25.3 | — | — | — | — | |
| BERTTKG background knowledge=without2024.02 | 7.1 | 8.6 | 5.2 | 7.7 | 6 | 21.3 | 20.5 | 22.5 | 19.2 | 25.3 | — | — | — | — | |
| RoBERTa2021.06 | 7 | 8.6 | 5 | 8.2 | 4.8 | 20.2 | 19.2 | 21.5 | 18.6 | 23.1 | — | — | — | — | |
| KnowBERT2021.06 | 7 | 8.3 | 5.1 | 8.1 | 4.8 | 20.1 | 18.9 | 21.7 | 18.5 | 23 | — | — | — | — | |
| RoBERTa2023.02 | 7 | 8.6 | 5 | 8.2 | 4.8 | 20.2 | 19.2 | 21.5 | 18.6 | 23.1 | — | — | — | — | |
| KnowBERT2023.02 | 7 | 8.3 | 5.1 | 8.1 | 4.8 | 20.1 | 18.9 | 21.7 | 18.5 | 23 | — | — | — | — | |
| RoBERTaTKG background knowledge=without2024.02 | 7 | 8.6 | 5 | 8.2 | 4.8 | 20.2 | 19.2 | 21.5 | 18.6 | 23.1 | — | — | — | — | |
| GPT-4o2026.02 | — | — | — | — | — | — | — | — | — | — | 20.6 | 6.9 | 13 | 37.6 | |
| LLaMA2-7B2026.02 | — | — | — | — | — | — | — | — | — | — | 16.9 | 4.9 | 14.3 | 28.2 | |
| RAG baseline2026.02 | — | — | — | — | — | — | — | — | — | — | 23.5 | 70.4 | 9.2 | 0.9 | |
| RTQA2026.02 | — | — | — | — | — | — | — | — | — | — | 29.8 | 60.8 | 21.8 | 13.5 | |
| TKG-ThinkerBackbone=Qwen2.5-7B-Instruct2026.02 | — | — | — | — | — | — | — | — | — | — | 46 | 56.7 | 29.4 | 52.3 | |
| TKG-ThinkerBackbone=Llama3-8B-Instruct2026.02 | — | — | — | — | — | — | — | — | — | — | 49.1 | 61.2 | 32 | 54.6 |