Table Fact Verification on TABFACT small (test)
0.921AccuracyHuman Performance
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Human Performance2020.10 | 0.921 | — | — | |
| Human Performance2021.07 | 0.921 | — | — | |
| Human Performance2022.11 | 0.921 | — | — | |
| PASTA2022.11 | 0.906 | — | — | |
| SaMoE2022.11 | 0.867 | — | — | |
| DeBERTaV32022.11 | 0.865 | — | — | |
| REASTAP2022.10 | 0.862 | — | — | |
| TAPEXpre-training_steps=50000, fine-tuning_steps=200002021.07 | 0.859 | — | — | |
| Tapex2022.11 | 0.859 | — | — | |
| Tab-PoTLLM=DeepSeek-67B2024.06 | 0.8577 | 91.34 | 80.27 | |
| DaterLLM=Codex2024.06 | 0.856 | 91.2 | 80 | |
| TAPEX2022.10 | 0.855 | — | — | |
| BinderLLM=Not Specified2024.06 | 0.851 | — | — | |
| DecompTaPas2022.10 | 0.847 | — | — | |
| TableFormer2022.10 | 0.846 | — | — | |
| TAPASModel=Large, Pre-training Objective=Counterfactual + Synthetic2020.10 | 0.839 | — | — | |
| Eisenschlos et al.model_category=Pre-trained Language Models2021.07 | 0.839 | — | — | |
| TaPas2022.10 | 0.839 | — | — | |
| Tapas2022.11 | 0.839 | — | — | |
| COTLLM=DeepSeek-67B2024.06 | 0.8261 | 88.46 | 76.84 | |
| BART2021.07 | 0.825 | — | — | |
| BART2022.10 | 0.823 | — | — | |
| Tab-PoTLLM=Mixtral-8x7B2024.06 | 0.8167 | 88.36 | 75.07 | |
| TAPASModel=Base, Pre-training Objective=Counterfactual + Synthetic2020.10 | 0.81 | — | — | |
| TAPASModel=Base, Pre-training Objective=Synthetic2020.10 | 0.804 | — | — | |
| COTLLM=Mixtral-8x7B2024.06 | 0.8009 | 86.07 | 74.19 | |
| Schlichtkrull et al.Retrieval=Oracle2022.11 | 0.794 | — | — | |
| DirectLLM=DeepSeek-67B2024.06 | 0.7861 | 84.68 | 72.62 | |
| TAPASModel=Base, Pre-training Objective=Counterfactual2020.10 | 0.774 | — | — | |
| TAPASModel=Base, Pre-training Objective=SQA2020.10 | 0.773 | — | — | |
| Yang et al.model_category=Pre-trained Language Models2021.07 | 0.762 | — | — | |
| ProgVGAT2022.11 | 0.762 | — | — | |
| DirectLLM=Mixtral-8x7B2024.06 | 0.7589 | 81.29 | 70.56 | |
| POTLLM=DeepSeek-67B2024.06 | 0.7451 | 76.32 | 72.72 | |
| LOGICALFACTCHECKER (program from Seq2Action)2020.10 | 0.743 | — | — | |
| Zhong et al.model_category=Pre-trained Language Models2021.07 | 0.743 | — | — | |
| LFC2022.10 | 0.743 | — | — | |
| LogicFactChecker2022.11 | 0.743 | — | — | |
| LOGICALFACTCHECKER (program from LPA)2020.10 | 0.742 | — | — | |
| Shi et al.model_category=Pre-trained Language Models2021.07 | 0.742 | — | — | |
| HeterTFV2022.10 | 0.742 | — | — | |
| POTLLM=Mixtral-8x7B2024.06 | 0.7332 | 76.02 | 70.66 | |
| TAPASModel=Base, Pre-training Objective=MASK-LM2020.10 | 0.722 | — | — | |
| Tab-PoTLLM=Mixtral-7B2024.06 | 0.7184 | 76.82 | 66.93 | |
| COTLLM=Mixtral-7B2024.06 | 0.7041 | 73.73 | 67.12 | |
| LPA-Ranking w/ Discriminator (Caption)2020.10 | 0.689 | — | — | |
| LPA-Ranking2022.10 | 0.689 | — | — | |
| TABLE-BERT-Horizontal-T+F-Template2020.10 | 0.681 | — | — | |
| Chen et al.model_category=Pre-trained Language Models2021.07 | 0.681 | — | — | |
| Table-BERT2022.11 | 0.681 | — | — | |
| DirectLLM=Mixtral-7B2024.06 | 0.6789 | 73.13 | 62.71 | |
| POTLLM=Mixtral-7B2024.06 | 0.6625 | 65.97 | 66.54 | |
| Tab-PoTLLM=DeepSeek-7B2024.06 | 0.6586 | 70.75 | 61.04 | |
| COTLLM=DeepSeek-7B2024.06 | 0.6517 | 70.65 | 59.76 | |
| POTLLM=DeepSeek-7B2024.06 | 0.6191 | 64.88 | 58.98 | |
| DirectLLM=DeepSeek-7B2024.06 | 0.5771 | 59.5 | 55.94 | |
| BERT classifier w/o Table2020.10 | 0.504 | — | — |