Factual Consistency Evaluation on TRUE benchmark

98.4PAWS (AUC-ROC)ALIGNSCORE-large

Evaluation Results

Method
ALIGNSCORE-large2023.05	98.4	82.9	78.6	98.3	94.9	92.1	85.1	76.1	89.5	83.5	82.5	87.4	83.8	—	—	—	—	—	—	—
ALIGNSCORE-base2023.05	97.3	80.8	76.1	97.8	94.6	90	83.1	79.9	87.7	79.6	82.4	86.3	82.5	—	—	—	—	—	—	—
M + A + W2024.02	89.95	—	76.45	88.55	—	—	—	79.49	—	—	83.56	85.69	—	—	90.6	87.23	86.73	79.44	85.56	95.07
SummaC-ZS2023.05	89	77.6	81.8	97.2	92.8	86.9	87.1	58	76	75.3	83.2	82.2	78.2	—	—	—	—	—	—	—
SummaC-CONV2023.05	88.2	79.1	77.5	97.5	92	89	81.2	67.2	77.7	76	81.6	82.5	78.7	—	—	—	—	—	—	—
WANLI2024.02	87.29	—	84.88	87.38	—	—	—	73.29	—	—	82.47	83.72	—	—	88.59	72.18	82.85	74.61	92.4	94.97
GNLI2024.02	87.1	—	77.4	87.69	—	—	—	79.97	—	—	84	85.75	—	—	91.38	85.48	87.03	79.28	88.57	95.34
ANLI2024.02	86.35	—	74.68	90.59	—	—	—	76.41	—	—	79.77	82.98	—	—	87.9	82.08	84.68	75.79	81.06	93.46
QAFactEval2023.05	86.1	80.9	75.8	73.6	86	88.5	81.8	67.3	83.9	76.1	81	80.1	79.4	—	—	—	—	—	—	—
M + A + W2024.02	85.86	—	79.81	86.48	—	—	—	78.28	—	—	81.13	82.2	—	—	83.57	72.28	82.27	72.61	87.25	94.63
ANLI2024.02	84.72	—	78.43	89.63	—	—	—	75.49	—	—	78.21	81.46	—	—	81.78	74.69	81.81	71.6	85.63	94.03
GNLI2024.02	84.63	—	79.77	85.86	—	—	—	79.49	—	—	83.45	84.72	—	—	90.14	81.33	84.02	79.75	88.76	94.73
WANLI2024.02	83.17	—	82.16	82.79	—	—	—	67.7	—	—	80.37	79.72	—	—	80.31	74.46	71.11	72.86	89.15	93.82
MNLI2024.02	83.11	—	76.84	84.58	—	—	—	78.31	—	—	81.76	82.79	—	—	88.18	79.03	83.07	72.35	88.32	95.13
MNLI2023.05	81.3	44.6	71.8	80.2	93.1	57.2	76.5	59.1	42.6	50.1	81.5	67.1	60.4	—	—	—	—	—	—	—
UniEval2023.05	80.1	81.2	70.4	79.1	92.1	88.1	80.4	66.8	86.5	76.7	73.6	79.5	78	—	—	—	—	—	—	—
BERTScore2023.05	78.6	72.3	70.2	58.2	54.2	84	68.6	52.5	70.6	44.3	86.4	67.2	68.6	—	—	—	—	—	—	—
BARTScore2023.05	77.1	78.9	65.1	64.2	66.1	87.8	60.8	63.5	83.9	60.2	86.7	72.2	73.4	—	—	—	—	—	—	—
ROUGE-L2023.05	75.9	80.4	60.6	59.7	56.4	83.6	65.4	62.8	77.6	59.3	85	69.7	71.8	—	—	—	—	—	—	—
MNLI2024.02	75.82	—	77.32	81.6	—	—	—	79.27	—	—	82.13	77.84	—	—	79.15	58.13	79.56	61.59	87.65	93.97
BLEU2023.05	71.3	74.8	55.2	56.1	51.7	84.1	61.2	56.7	77.4	54.7	74.6	65.2	67.3	—	—	—	—	—	—	—
SimCSE2023.05	69.2	70.2	66.2	63.8	72.7	72.9	70.6	64.6	74.9	56.5	86.1	69.8	70.3	—	—	—	—	—	—	—
QuestEval2023.05	69	69.7	72.2	66.6	72.5	84	77.2	64.8	64.5	55.2	83.9	70.9	71.4	—	—	—	—	—	—	—
ROUGE-22023.05	68.6	79.4	61.4	59.9	55.5	84.5	67.7	65	78.4	60.2	82.8	69.4	72.4	—	—	—	—	—	—	—
BLEURT2023.05	68.4	68	72.9	61.8	59.5	81.6	73	65.5	71.2	56.2	86.6	69.5	71.9	—	—	—	—	—	—	—
CTC2023.05	63.1	79.8	66.8	65	72.5	87.1	63.7	65	77.3	67.7	72	70.9	72.4	—	—	—	—	—	—	—
M + A + W2024.02	59.15	—	66	72.71	—	—	—	69.69	—	—	79.08	65.33	—	—	50.2	47.35	61.76	46.75	76.01	89.94
GNLI2024.02	57.33	—	72.1	76.11	—	—	—	76.91	—	—	82.21	72.06	—	—	67.32	60.22	72.39	56.29	81.23	90.54
BLANC2023.05	56	63.3	62.9	55.7	53.6	82.1	63.8	54.2	60.9	50.9	73.7	61.6	64	—	—	—	—	—	—	—
DAE2023.05	55.8	60.3	57.7	60.2	77.8	77.9	54.7	81	56.9	67.5	69.4	65.4	65.7	—	—	—	—	—	—	—
FactCC2023.05	53.4	68.6	59.3	54.7	58.7	70.7	55	56.1	70.1	64.4	57.6	60.8	62.7	—	—	—	—	—	—	—
NER-Overlap2023.05	51.7	56.6	59.1	57.8	62.4	65.5	62.7	68.4	48.4	63.6	50.6	58.8	59.3	—	—	—	—	—	—	—
MNLI2024.02	51.68	—	68.4	70.1	—	—	—	70.3	—	—	80.77	63.48	—	—	49.62	37.76	58.3	45.97	76.1	89.33
WANLI2024.02	51.51	—	68.97	67.85	—	—	—	69.9	—	—	65.79	65.21	—	—	57.99	54.14	70.21	48.98	77.62	84.35
ROUGE-12023.05	50.2	80.4	59.7	60.9	57.8	83.6	65.3	64.8	77.3	60.1	84.6	67.7	72	—	—	—	—	—	—	—
FEQA2023.05	50	49.5	53.2	49.9	51.1	63	50.5	48.8	50.1	49.4	53	51.7	52.2	—	—	—	—	—	—	—
ANLI2024.02	47.09	—	45.02	53.5	—	—	—	53.99	—	—	57.66	51.77	—	—	50.95	54.64	44.7	51.34	55.39	55.24
All2023.05	—	—	—	—	—	—	—	—	—	—	—	—	—	83.2	—	—	—	—	—	—
All (-MC)2023.05	—	—	—	—	—	—	—	—	—	—	—	—	—	82.7	—	—	—	—	—	—
Q22023.05	—	—	—	—	—	—	—	—	—	—	—	—	—	81.4	—	—	—	—	—	—
SummacZS2023.05	—	—	—	—	—	—	—	—	—	—	—	—	—	80.7	—	—	—	—	—	—
T5 ANLI2023.05	—	—	—	—	—	—	—	—	—	—	—	—	—	81.5	—	—	—	—	—	—