Fact Verification on FEVER
53.9AccuracyCOT-SC + VE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| COT-SC + VEknowledge=Google, Model=GPT-3 davinci-0032023.05 | 53.9 | 1.9 | |
| COT-SC + VEknowledge=Wiki., Model=GPT-3 davinci-0032023.05 | 53.6 | 1.6 | |
| COT-SC + VEknowledge=DRQA, Model=GPT-3 davinci-0032023.05 | 53.3 | 1.3 | |
| COT-SCknowledge=-, Model=GPT-3 davinci-0032023.05 | 52 | — | |
| COTknowledge=-, Model=GPT-3 davinci-0032023.05 | 50 | — | |
| Standardknowledge=-, Model=GPT-3 davinci-0032023.05 | 46.8 | — | |
| CoT-SC + Calib.knowledge=-, Model=GPT-3 davinci-0032023.05 | 33.7 | — | |
| Llama3-RankRAG 70BRetrieval-Augmented Generation=Enabled, Zero-shot=true2024.07 | 0.938 | — | |
| Llama3-ChatQA-1.5 70BRetrieval-Augmented Generation=Enabled2024.07 | 0.927 | — | |
| GPT-4-0613 RAGRetrieval-Augmented Generation=Enabled2024.07 | 0.926 | — | |
| GPT-4-turbo-2024-0409 RAGRetrieval-Augmented Generation=Enabled2024.07 | 0.922 | — | |
| Llama3-RankRAG 8BRetrieval-Augmented Generation=Enabled, Zero-shot=true2024.07 | 0.92 | — | |
| Llama3-Instruct 70BRetrieval-Augmented Generation=Enabled2024.07 | 0.914 | — | |
| Llama3-ChatQA-1.5 8BRetrieval-Augmented Generation=Enabled2024.07 | 0.909 | — | |
| GPT-3.5-turbo-1106 RAGRetrieval-Augmented Generation=Enabled2024.07 | 0.908 | — | |
| Supervised SoTA2022.10 | 0.895 | — | |
| PROMPTEDPrompting Technique=PROMPTED2023.10 | 0.892 | — | |
| Llama3-Instruct 8BRetrieval-Augmented Generation=Enabled2024.07 | 0.889 | — | |
| GPT-4-0613Retrieval-Augmented Generation=Disabled2024.07 | 0.877 | — | |
| Output RefinementPrompting Technique=Output Refinement2023.10 | 0.876 | — | |
| GPT-4-turbo-2024-0409Retrieval-Augmented Generation=Disabled2024.07 | 0.87 | — | |
| Zero-Shot CoTPrompting Technique=Zero-Shot Chain-of-Thought2023.10 | 0.868 | — | |
| GPT-3.5-turbo-1106Retrieval-Augmented Generation=Disabled2024.07 | 0.827 | — | |
| ReFeedRetrieval-Augmented Generation=Enabled2024.07 | 0.827 | — | |
| PlugDPost-processing=+DPlug2023.05 | 0.8254 | — | |
| RA-DIT 65BRetrieval-Augmented Generation=Enabled2024.07 | 0.807 | — | |
| GenReadRetrieval-Augmented Generation=Enabled2024.07 | 0.804 | — | |
| PlugDPost-processing=None2023.05 | 0.7956 | — | |
| ACPSModel=GPT-3.5-turbo2026.01 | 0.7948 | — | |
| T5Post-processing=None2023.05 | 0.791 | — | |
| Zero-ShotPrompting Technique=Zero-Shot2023.10 | 0.788 | — | |
| InstructGPTRetrieval-Augmented Generation=Disabled2024.07 | 0.776 | — | |
| CPModel=GPT-3.5-turbo2026.01 | 0.7713 | — | |
| Atlas 11BRetrieval-Augmented Generation=Enabled2024.07 | 0.77 | — | |
| T5Post-processing=+Concat2023.05 | 0.7684 | — | |
| RePlug 65BRetrieval-Augmented Generation=Enabled2024.07 | 0.733 | — | |
| MOCHAEvaluation Backbone=Claude Haiku 4.5, Rollout Budget=1000, Number of Seeds=52026.05 | 0.726 | — | |
| ACPSModel=LLaMA-32026.01 | 0.6515 | — | |
| DeCoTModel=GPT-3.5-turbo2026.01 | 0.6469 | — | |
| CoT-SC -> ReActBackbone=PaLM-540B2022.10 | 0.646 | — | |
| CADModel=GPT-3.5-turbo2026.01 | 0.6453 | — | |
| CPModel=LLaMA-32026.01 | 0.6364 | — | |
| Seed SkillEvaluation Backbone=Claude Haiku 4.5, Rollout Budget=1000, Number of Seeds=52026.05 | 0.632 | — | |
| TextGradEvaluation Backbone=Claude Haiku 4.5, Rollout Budget=1000, Number of Seeds=52026.05 | 0.632 | — | |
| ProTeGiEvaluation Backbone=Claude Haiku 4.5, Rollout Budget=1000, Number of Seeds=52026.05 | 0.632 | — | |
| GEPAEvaluation Backbone=Claude Haiku 4.5, Rollout Budget=1000, Number of Seeds=52026.05 | 0.632 | — | |
| ReAct -> CoT-SCBackbone=PaLM-540B2022.10 | 0.62 | — | |
| ReActBackbone=PaLM-540B2022.10 | 0.609 | — | |
| CoT-SCBackbone=PaLM-540B2022.10 | 0.604 | — | |
| ACPSModel=Ministral-3B2026.01 | 0.6 | — | |
| ActBackbone=PaLM-540B2022.10 | 0.589 | — | |
| DeCoTModel=LLaMA-32026.01 | 0.5752 | — | |
| StandardBackbone=PaLM-540B2022.10 | 0.571 | — | |
| CoTBackbone=PaLM-540B2022.10 | 0.563 | — | |
| CoT-SCModel=GPT-3.5-turbo2026.01 | 0.5462 | — | |
| CPModel=Ministral-3B2026.01 | 0.5455 | — | |
| CADModel=LLaMA-32026.01 | 0.5355 | — | |
| CoT-SCModel=LLaMA-32026.01 | 0.5302 | — | |
| CoTModel=LLaMA-32026.01 | 0.5245 | — | |
| SoTModel=LLaMA-32026.01 | 0.522 | — | |
| SoTModel=GPT-3.5-turbo2026.01 | 0.4821 | — | |
| CoTModel=GPT-3.5-turbo2026.01 | 0.4706 | — | |
| ICLModel=GPT-3.5-turbo2026.01 | 0.4615 | — | |
| ICLModel=LLaMA-32026.01 | 0.4582 | — | |
| CoT-SCModel=Ministral-3B2026.01 | 0.4167 | — | |
| DeCoTModel=Ministral-3B2026.01 | 0.4158 | — | |
| CADModel=Ministral-3B2026.01 | 0.3965 | — | |
| RECOMP 20BRetrieval-Augmented Generation=Enabled2024.07 | 0.389 | — | |
| InstructRetro 43BRetrieval-Augmented Generation=Enabled2024.07 | 0.36 | — | |
| CoTModel=Ministral-3B2026.01 | 0.3158 | — | |
| SoTModel=Ministral-3B2026.01 | 0.3115 | — | |
| ICLModel=Ministral-3B2026.01 | 0.2941 | — | |
| COT-SC → ReActknowledge=Wiki., Model=PaLM2023.05 | — | 4.2 | |
| ReAct → COT-SCknowledge=Wiki., Model=PaLM2023.05 | — | 1.6 |