List your company
ResearchFormerly WizWand

Knowledge Base Question Answering on GrailQA (test)

91.76F1Pangu (T5-base)
12.210432.862753.51574.1673Nov 15, 2023Dec 19, 2023Jan 23, 2024Feb 27, 2024Apr 2, 2024May 7, 2024Jun 11, 2024
Updated 8mo ago

Evaluation results

MethodLinks
Pangu (T5-base)Learning Mode=Bottom-up Parser, Data=full train data, Backbone=T5-base, Zero-shot=true, Entity Linker=Golden2024.0691.7688.3
ArcaneQALearning Mode=Bottom-up Parser, Data=full train data, Zero-shot=true, Entity Linker=Golden2024.0681.8178.52
DARA (w/Llama-2-13B)Learning Mode=Fine-tuned LLM Agent, Backbone=Llama-2-13B, Zero-shot=true, Entity Linker=Golden2024.0680.3577.03
DARA (w/Mistral-7B)Learning Mode=Fine-tuned LLM Agent, Backbone=Mistral-7B, Zero-shot=true, Entity Linker=Golden2024.0680.1676.88
DARA (w/CodeLlama-7B)Learning Mode=Fine-tuned LLM Agent, Backbone=CodeLlama-7B, Zero-shot=true, Entity Linker=Golden2024.0678.6175.26
DARA (w/Llama-2-7B)Learning Mode=Fine-tuned LLM Agent, Backbone=Llama-2-7B, Zero-shot=true, Entity Linker=Golden2024.0677.7175.05
FuSIC-KBQAEvaluation Protocol=Few-shot, Category=Fusion, Retriever=TIARA and Pangu2023.1169.1—
FuSIC-KBQAEvaluation Protocol=Zero-shot, Category=Fusion, Retriever=TIARA and Pangu2023.1167.5—
AgentBench (GPT-4)Learning Mode=Off-the-shelf (in-context learning), Backbone=GPT-4, Zero-shot=true, Entity Linker=Golden2024.0665.8963.56
FuSIC-KBQAEvaluation Protocol=Few-shot, Category=Fusion, Retriever=Pangu2023.1164.5—
FuSIC-KBQAEvaluation Protocol=Zero-shot, Category=Fusion, Retriever=Pangu2023.1163.8—
PanguEvaluation Protocol=Few-shot, Category=Supervised2023.1163.2—
PanguEvaluation Protocol=Zero-shot, Category=Supervised2023.1160.9—
FuSIC-KBQAEvaluation Protocol=Few-shot, Category=Fusion, Retriever=TIARA2023.1160.3—
AgentBench-7BLearning Mode=Fine-tuned LLM Agent, Backbone=7B, Zero-shot=true, Entity Linker=Golden2024.0659.2856.96
FuSIC-KBQAEvaluation Protocol=Zero-shot, Category=Fusion, Retriever=TIARA2023.1159.2—
AgentLM-13BLearning Mode=Fine-tuned LLM Agent, Backbone=13B, Zero-shot=true, Entity Linker=Golden2024.0655.0152.72
gf-LLMEvaluation Protocol=Zero-shot, Category=LLM ICL2023.1153.4—
TIARAEvaluation Protocol=Zero-shot, Category=Supervised2023.1152.8—
gf-LLMEvaluation Protocol=Few-shot, Category=LLM ICL2023.1152.5—
TIARAEvaluation Protocol=Few-shot, Category=Supervised2023.1150.1—
KB-BinderEvaluation Protocol=Zero-shot, Category=LLM ICL2023.1145—
BERT-RankingEvaluation Protocol=Few-shot, Category=LLM ICL2023.1144.1—
BERT-RankingEvaluation Protocol=Zero-shot, Category=LLM ICL2023.1142.4—
AgentBench (Llama-2-chat-70B)Learning Mode=Off-the-shelf (in-context learning), Backbone=Llama-2-chat-70B, Zero-shot=true, Entity Linker=Golden2024.0635.7233.2
KB-BinderEvaluation Protocol=Few-shot, Category=LLM ICL2023.1135.2—
AgentLM-7BLearning Mode=Fine-tuned LLM Agent, Backbone=7B, Zero-shot=true, Entity Linker=Golden2024.0615.2714.45
Pangu (T5-Large)Learning Mode=Bottom-up Parser, Data=full train data, Backbone=T5-Large, Zero-shot=true, Entity Linker=Golden2024.06—67.21