Factuality Evaluation on LLM-AggreFact (test)
69.9CNN ScoreMiniCheck-FT5
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MiniCheck-FT5Threshold Tuning=False2024.10 | 69.9 | 74.3 | 73.6 | 77.3 | 72.2 | 86.2 | 74.6 | 74.7 | 59 | 85.2 | 74.7 | — | |
| GPT-4Threshold Tuning=False2024.10 | 66.7 | 76.5 | 71.4 | 79.9 | 80.4 | 87.8 | 67.6 | 79.9 | 59.2 | 83.1 | 75.3 | — | |
| Claude-3 OpusThreshold Tuning=False2024.10 | 65.2 | 72.4 | 74.1 | 82.4 | 75 | 83.8 | 69.3 | 78.8 | 58.8 | 81.6 | 74.1 | — | |
| Llama3-70B-chatThreshold Tuning=False2024.10 | 63.3 | 71.3 | 67.9 | 75.2 | 74.8 | 86.7 | 67.3 | 78.3 | 58.4 | 82.9 | 72.6 | — | |
| FENCE (Vanilla SFT)Threshold Tuning=False, Backbone=Llama3-8B, Training Protocol=Vanilla SFT2024.10 | 63.2 | 73.4 | 66.6 | 77.7 | 64.6 | 86.4 | 72.5 | 73 | 57.9 | 83.1 | 71.8 | — | |
| GPT-3.5Threshold Tuning=False2024.10 | 63.2 | 72.4 | 66.8 | 73.4 | 68.5 | 84.7 | 65.2 | 70.8 | 57.2 | 73.8 | 69.6 | — | |
| FENCE (Full)Threshold Tuning=False, Backbone=Llama3-8B, Training Protocol=Augmented Critique and Source Documents2024.10 | 62.1 | 72.4 | 70.9 | 80.3 | 76 | 88.6 | 74.9 | 74.4 | 60.3 | 86.9 | 74.7 | — | |
| Claude-2.1Threshold Tuning=False2024.10 | 59.9 | 66.4 | 69.2 | 72.3 | 64.3 | 88.2 | 69.7 | 79.3 | 59.8 | 78.2 | 70.7 | — | |
| FENCE (Critique Only)Threshold Tuning=False, Backbone=Llama3-8B, Training Protocol=Critique Only2024.10 | 59.5 | 74.7 | 68.4 | 80 | 71.7 | 88 | 74.3 | 74.2 | 59.6 | 87 | 73.7 | — | |
| Mistral-123BThreshold Tuning=False2024.10 | 58.4 | 76.3 | 67.3 | 78.9 | 76.6 | 88.4 | 67.6 | 79 | 60 | 81.7 | 73.4 | — | |
| Mistral-8x7BThreshold Tuning=False2024.10 | 55 | 65.5 | 68.5 | 73.3 | 63.8 | 80.8 | 64.3 | 75.1 | 56.3 | 70.8 | 67.3 | — | |
| Llama3-8B-chatThreshold Tuning=False, Backbone=Llama3-8B2024.10 | 50.9 | 58.2 | 63.9 | 72.4 | 65.1 | 85.2 | 63.8 | 76.5 | 55.9 | 72.3 | 66.4 | — | |
| Gemini-ProThreshold Tuning=False2024.10 | 49.4 | 60.6 | 63.8 | 65.8 | 65.8 | 85.5 | 61.8 | 76.8 | 56.8 | 75.9 | 66.2 | — | |
| MiniCheck-7BModel Size=7B2026.04 | — | — | — | — | — | — | — | — | — | — | — | 77.4 | |
| ThinknCheck-1BModel Size=1B2026.04 | — | — | — | — | — | — | — | — | — | — | — | 78.1 | |
| ThinknCheck-nothink-1BModel Size=1B, Reasoning=None2026.04 | — | — | — | — | — | — | — | — | — | — | — | 57.5 | |
| ThinknCheck-Science-1BModel Size=1B, Training Data=Scientific and arithmetic2026.04 | — | — | — | — | — | — | — | — | — | — | — | 79.2 |