List your company
ResearchFormerly WizWand

Factuality Detection on Short-form QA (Average of NQ, PopQA, TriviaQA, SimpleQA) (test)

71.1PR-AUCFRANQ condition-calibrated
35.01244.38153.7563.119May 27, 2025
Updated 6mo ago

Evaluation results

MethodLinks
FRANQ condition-calibratedLLM Model=Falcon 3B Base2025.0571.147.7
XGBoost (all UQ features)LLM Model=Falcon 3B Base2025.0570.546.2
Degree MatrixLLM Model=Falcon 3B Base2025.0570.246.4
Sum of EigenvaluesLLM Model=Falcon 3B Base2025.057046
FRANQ calibratedLLM Model=Falcon 3B Base2025.0567.241.1
XGBoost (FRANQ features)LLM Model=Falcon 3B Base2025.056736.8
AlignScoreLLM Model=Falcon 3B Base2025.0566.637.2
FRANQ condition-calibratedLLM Model=Llama 8B Instruct2025.0564.754
FRANQ calibratedLLM Model=Llama 8B Instruct2025.0564.453.4
CCPLLM Model=Falcon 3B Base2025.0564.130.4
FRANQ no calibrationLLM Model=Falcon 3B Base2025.0564.134.5
Mean Token EntropyLLM Model=Llama 8B Instruct2025.056449.1
Lexical SimilarityLLM Model=Llama 8B Instruct2025.0563.953.2
Semantic EntropyLLM Model=Llama 8B Instruct2025.0563.751.9
XGBoost (all UQ features)LLM Model=Llama 8B Instruct2025.0563.450.3
FRANQ condition-calibratedLLM Model=Llama 3B Instruct2025.0563.154.1
Degree MatrixLLM Model=Llama 3B Instruct2025.0562.952
Max Sequence Prob.LLM Model=Falcon 3B Base2025.0562.825.6
Sum of EigenvaluesLLM Model=Llama 3B Instruct2025.0562.851.8
Sum of EigenvaluesLLM Model=Llama 8B Instruct2025.0562.848.9
FRANQ calibratedLLM Model=Llama 3B Instruct2025.0562.853.7
Degree MatrixLLM Model=Llama 8B Instruct2025.0562.749.2
Semantic EntropyLLM Model=Falcon 3B Base2025.0562.327.8
Lexical SimilarityLLM Model=Falcon 3B Base2025.0561.827.7
Mean Token EntropyLLM Model=Falcon 3B Base2025.0561.324.2
Semantic EntropyLLM Model=Llama 3B Instruct2025.0561.352.5
SentenceSARLLM Model=Falcon 3B Base2025.0560.226.3
Mean Token EntropyLLM Model=Llama 3B Instruct2025.0559.448.1
XGBoost (all UQ features)LLM Model=Llama 3B Instruct2025.0559.449.4
SentenceSARLLM Model=Llama 3B Instruct2025.0557.148.3
Max Sequence Prob.LLM Model=Llama 8B Instruct2025.0556.940.7
Lexical SimilarityLLM Model=Llama 3B Instruct2025.0556.447.9
Max Sequence Prob.LLM Model=Llama 3B Instruct2025.0555.845.4
SentenceSARLLM Model=Llama 8B Instruct2025.0555.641.4
Parametric KnowledgeLLM Model=Falcon 3B Base2025.0555.610.4
CCPLLM Model=Llama 8B Instruct2025.0555.341.7
FRANQ no calibrationLLM Model=Llama 3B Instruct2025.0555.340.3
CCPLLM Model=Llama 3B Instruct2025.0555.144.3
XGBoost (FRANQ features)LLM Model=Llama 3B Instruct2025.0552.640.9
XGBoost (FRANQ features)LLM Model=Llama 8B Instruct2025.0552.438.5
FRANQ no calibrationLLM Model=Llama 8B Instruct2025.0552.334
Parametric KnowledgeLLM Model=Llama 8B Instruct2025.0549.933
FRANQ condition-calibratedLLM Model=Gemma 12B Instruct2025.0549.628.3
FRANQ calibratedLLM Model=Gemma 12B Instruct2025.0548.125.8
XGBoost (all UQ features)LLM Model=Gemma 12B Instruct2025.0547.430.1
Sum of EigenvaluesLLM Model=Gemma 12B Instruct2025.0546.726
Semantic EntropyLLM Model=Gemma 12B Instruct2025.0546.626.1
Degree MatrixLLM Model=Gemma 12B Instruct2025.0546.426
FRANQ no calibrationLLM Model=Gemma 12B Instruct2025.0544.722.5
AlignScoreLLM Model=Llama 8B Instruct2025.0543.222.4
Lexical SimilarityLLM Model=Gemma 12B Instruct2025.054324
Parametric KnowledgeLLM Model=Llama 3B Instruct2025.0542.524.7
Mean Token EntropyLLM Model=Gemma 12B Instruct2025.0542.323
SentenceSARLLM Model=Gemma 12B Instruct2025.0541.617.4
AlignScoreLLM Model=Llama 3B Instruct2025.0541.520.7
XGBoost (FRANQ features)LLM Model=Gemma 12B Instruct2025.0541.419.6
CCPLLM Model=Gemma 12B Instruct2025.0541.219.8
Max Sequence Prob.LLM Model=Gemma 12B Instruct2025.054016.2
AlignScoreLLM Model=Gemma 12B Instruct2025.0537.615.8
Parametric KnowledgeLLM Model=Gemma 12B Instruct2025.0536.410.5