Knowledge Base Question Answering on GrailQA v1.0 (test)
77.45Overall EMRetinaQA
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| RetinaQATrain=A+U, Thresholding=true2024.03 | 77.45 | — | — | — | — | — | — | — | 83.3 | 82.69 | |
| PanguBackbone=T5-3B2022.12 | 75.4 | 81.7 | 84.4 | 88.8 | 74.6 | 81.5 | 71.6 | 78.5 | — | — | |
| PanguBackbone=T5-large2022.12 | 74.8 | 81.4 | 82.5 | 87.3 | 75.2 | 82.2 | 71 | 78.4 | — | — | |
| RetinaQATrain=A, Thresholding=true2024.03 | 73.76 | — | — | — | — | — | — | — | 81.8 | 80.78 | |
| PanguBackbone=BERT-base2022.12 | 73.7 | 79.9 | 82.6 | 87.1 | 74.9 | 81.2 | 69.1 | 76.1 | — | — | |
| PanguBackbone=T5-base2022.12 | 73.6 | 79.9 | 84.7 | 88.8 | 73.1 | 80.1 | 68.6 | 75.8 | — | — | |
| TIARA2022.12 | 73 | 78.5 | 87.8 | 90.6 | 69.2 | 76.5 | 68 | 73.9 | — | — | |
| Uni-Parser2022.12 | 69.5 | 74.6 | 85.5 | 88.5 | 65.1 | 71.1 | 64 | 69.8 | — | — | |
| RnG-KBQA2022.12 | 68.8 | 74.4 | 86.2 | 89 | 63.8 | 71.2 | 63 | 69.2 | — | — | |
| DecAF2022.12 | 68.4 | 78.7 | 84.8 | 89.9 | 73.4 | 81.8 | 58.6 | 72.3 | — | — | |
| RnG-KBQATrain=A+U, Thresholding=false2024.03 | 68.2 | — | — | — | — | — | — | — | 80.5 | 79.4 | |
| RnG-KBQATrain=A+U, Thresholding=true2024.03 | 67.8 | — | — | — | — | — | — | — | 77.8 | 77.1 | |
| TIARATrain=A+U, Thresholding=true2024.03 | 66.87 | — | — | — | — | — | — | — | 77.67 | 76.94 | |
| PanguTrain=A, Thresholding=true2024.03 | 66.53 | — | — | — | — | — | — | — | 79.14 | 77.89 | |
| TIARATrain=A+U, Thresholding=false2024.03 | 66.29 | — | — | — | — | — | — | — | 78.29 | 77.43 | |
| PanguTrain=A+U, Thresholding=true2024.03 | 66.14 | — | — | — | — | — | — | — | 78.29 | 76.91 | |
| RetinaQA - LFRTrain=A+U, Thresholding=true, Ablation=minus logical form retriever2024.03 | 65.37 | — | — | — | — | — | — | — | 77.36 | 76.37 | |
| RetinaQATrain=A+U, Thresholding=false2024.03 | 64.79 | — | — | — | — | — | — | — | 77.31 | 75.71 | |
| RetinaQATrain=A, Thresholding=false2024.03 | 64.54 | — | — | — | — | — | — | — | 76.83 | 75.24 | |
| ArcaneQA2022.12 | 63.8 | 73.7 | 85.6 | 88.9 | 65.8 | 75.3 | 52.9 | 66 | — | — | |
| ReTraCk2022.12 | 58.1 | 65.3 | 84.4 | 87.5 | 61.5 | 70.9 | 44.6 | 52.5 | — | — | |
| RnG-KBQATrain=A, Thresholding=true2024.03 | 57 | — | — | — | — | — | — | — | 67.6 | 65.8 | |
| PanguBackbone=Codex, Shots=10002022.12 | 56.4 | 65 | 67.5 | 73.7 | 58.2 | 64.9 | 50.7 | 61.1 | — | — | |
| TIARATrain=A, Thresholding=true2024.03 | 55.23 | — | — | — | — | — | — | — | 73.26 | 71.62 | |
| PanguTrain=A, Thresholding=false2024.03 | 54.55 | — | — | — | — | — | — | — | 63.09 | 60.06 | |
| RetinaQA - LFITrain=A+U, Thresholding=true, Ablation=minus logical form integrator2024.03 | 53.89 | — | — | — | — | — | — | — | 74.89 | 73.53 | |
| PanguTrain=A+U, Thresholding=false2024.03 | 53.79 | — | — | — | — | — | — | — | 63.59 | 60.42 | |
| TIARATrain=A, Thresholding=false2024.03 | 53.69 | — | — | — | — | — | — | — | 75.05 | 72.84 | |
| PanguBackbone=Codex, Shots=1002022.12 | 53.3 | 62.7 | 54.7 | 62.9 | 54.5 | 63.7 | 52.3 | 62.2 | — | — | |
| RetinaQA - (SG U SR)Train=A+U, Thresholding=true, Ablation=minus sketch generator and schema retriever together2024.03 | 52.46 | — | — | — | — | — | — | — | 64.68 | 62.58 | |
| RnG-KBQATrain=A, Thresholding=false2024.03 | 51.6 | — | — | — | — | — | — | — | 67.8 | 65.6 | |
| BERT+Ranking2022.12 | 50.6 | 58 | 59.9 | 67 | 45.5 | 53.9 | 48.6 | 55.7 | — | — | |
| PanguBackbone=Codex, Shots=102022.12 | 48.9 | 56.3 | 51.8 | 58.1 | 43.3 | 51.2 | 50.1 | 57.8 | — | — | |
| QGG2022.12 | — | 36.7 | — | 40.5 | — | 33 | — | 36.6 | — | — | |
| UnifiedSKGBackbone=T5-3B, Oracle Entity Linking=true2022.12 | — | 70.1 | — | — | — | — | — | — | — | — |