Citation Hallucination Detection on NeuCLIR 2024 (test)
73.7AUCFACTUM
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| FACTUMBase Model=Llama-3.1-8B-Instruct, Classifier=Log. Reg.2026.01 | 73.7 | 0.33 | 33.4 | 68.8 | 44.6 | |
| FACTUMBase Model=Llama-3.1-8B-Instruct, Classifier=EBM2026.01 | 73.6 | 0.326 | 32.2 | 66.9 | 43.2 | |
| FACTUMBase Model=Llama-3.2-3B-Instruct, Classifier=Log. Reg.2026.01 | 71.5 | 0.354 | 48.4 | 69.3 | 56.5 | |
| FACTUMBase Model=Llama-3.2-3B-Instruct, Classifier=EBM2026.01 | 70 | 0.341 | 47.3 | 59.1 | 51.5 | |
| FACTUMBase Model=Llama-3.1-8B-Instruct, Classifier=LightGBM2026.01 | 69.3 | 0.269 | 30 | 65.2 | 40.8 | |
| CAS+PFSBase Model=Llama-3.1-8B-Instruct, Classifier=EBM2026.01 | 68.7 | 0.255 | 27.8 | 66 | 38.8 | |
| FACTUMBase Model=Llama-3.2-3B-Instruct, Classifier=LightGBM2026.01 | 68.2 | 0.302 | 43.6 | 62.4 | 50.7 | |
| CAS+PFSBase Model=Llama-3.1-8B-Instruct, Classifier=Log. Reg.2026.01 | 67.7 | 0.243 | 27.5 | 63.7 | 38.1 | |
| CAS+PFSBase Model=Llama-3.2-3B-Instruct, Classifier=Log. Reg.2026.01 | 67.6 | 0.282 | 43.1 | 66.7 | 51.9 | |
| LN-EntropyBase Model=Llama-3.1-8B-Instruct, Classifier=Classifier-free2026.01 | 67.1 | 0.219 | 23.7 | 62.5 | 33.3 | |
| PerplexityBase Model=Llama-3.1-8B-Instruct, Classifier=Classifier-free2026.01 | 67 | 0.137 | 23.7 | 62.9 | 33.4 | |
| CAS+PFSBase Model=Llama-3.2-3B-Instruct, Classifier=EBM2026.01 | 66.8 | 0.272 | 42.9 | 58.7 | 48.8 | |
| ECS+PKSBase Model=Llama-3.2-3B-Instruct, Classifier=Log. Reg.2026.01 | 66.8 | 0.272 | 40.6 | 63.6 | 49.1 | |
| CAS+PFSBase Model=Llama-3.2-3B-Instruct, Classifier=LightGBM2026.01 | 64.5 | 0.237 | 42.6 | 60.4 | 49.1 | |
| PerplexityBase Model=Llama-3.2-3B-Instruct, Classifier=Classifier-free2026.01 | 63.8 | 0.085 | 37.3 | 73.9 | 48.9 | |
| LN-EntropyBase Model=Llama-3.2-3B-Instruct, Classifier=Classifier-free2026.01 | 63.7 | 0.189 | 37.6 | 71.7 | 48.7 | |
| EnergyBase Model=Llama-3.1-8B-Instruct, Classifier=Classifier-free2026.01 | 61.7 | 0.14 | 21.9 | 41.9 | 26.9 | |
| ECS+PKSBase Model=Llama-3.1-8B-Instruct, Classifier=EBM2026.01 | 61.3 | 0.151 | 24.2 | 57.1 | 33.7 | |
| ECS+PKSBase Model=Llama-3.2-3B-Instruct, Classifier=EBM2026.01 | 61 | 0.18 | 39.1 | 58.3 | 46.3 | |
| EnergyBase Model=Llama-3.2-3B-Instruct, Classifier=Classifier-free2026.01 | 60.6 | 0.151 | 37.3 | 56 | 43.7 | |
| CAS+PFSBase Model=Llama-3.1-8B-Instruct, Classifier=LightGBM2026.01 | 59.2 | 0.124 | 23.6 | 58.3 | 33.3 | |
| ECS+PKSBase Model=Llama-3.2-3B-Instruct, Classifier=LightGBM2026.01 | 58.8 | 0.149 | 38.8 | 55.3 | 45 | |
| ECS+PKSBase Model=Llama-3.1-8B-Instruct, Classifier=LightGBM2026.01 | 54.9 | 0.07 | 21.2 | 53.4 | 30.2 | |
| ECS+PKSBase Model=Llama-3.1-8B-Instruct, Classifier=Log. Reg.2026.01 | 53.6 | 0.049 | 20.1 | 52.9 | 29 | |
| P(True)Base Model=Llama-3.2-3B-Instruct, Classifier=Classifier-free2026.01 | 48.1 | -0.035 | 27.8 | 58.8 | 36.9 | |
| P(True)Base Model=Llama-3.1-8B-Instruct, Classifier=Classifier-free2026.01 | 48.1 | -0.019 | 15.5 | 91.5 | 26.1 |