Factual Consistency Evaluation on SummaC
88.4CGSALIGN
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| ALIGNModel Scale=large2023.07 | 88.4 | 74.6 | 92.5 | 94.9 | 92.3 | 91.3 | 89 | |
| ALIGNSCORE-largeType=Ours2023.05 | 86.4 | 75.8 | 92.4 | 93.7 | 91.7 | 91.4 | 88.6 | |
| UniEvalType=Misc2023.05 | 84.7 | 65.5 | 93.4 | 89.9 | 86.3 | 88 | 84.6 | |
| UniEval2023.07 | 84.7 | 65.5 | 93.4 | 89.9 | 86.3 | 88 | 84.6 | |
| ALIGNSCORE-baseType=Ours2023.05 | 83.7 | 79.4 | 87.8 | 93.3 | 89.9 | 90.5 | 87.4 | |
| QAFactEvalType=QA2023.05 | 83.4 | 66.1 | 86.4 | 89.2 | 88.1 | 89.4 | 83.8 | |
| QAFactEval2023.07 | 83.4 | 66.1 | 86.4 | 89.2 | 88.1 | 89.4 | 83.8 | |
| ALIGNModel Scale=base2023.07 | 80.6 | 76.1 | 87.5 | 93.1 | 88.6 | 89.5 | 85.9 | |
| ALIGNSCORE-baseType=Ours2023.05 | 77.8 | 72.2 | 78.9 | 87.4 | 83.7 | 83.6 | 80.6 | |
| UniEvalType=Misc2023.05 | 77.1 | 61.2 | 85.3 | 84.7 | 79.4 | 80.9 | 78.1 | |
| CTCType=Misc2023.05 | 76.5 | 65.9 | 89.5 | 82.6 | 85.6 | 87.3 | 81.2 | |
| CTC2023.07 | 76.5 | 65.9 | 89.5 | 82.6 | 85.6 | 87.3 | 81.2 | |
| ALIGNSCORE-largeType=Ours2023.05 | 75 | 70 | 88 | 89.2 | 83.4 | 86.3 | 82 | |
| BARTScoreType=Misc2023.05 | 74.3 | 62.6 | 91.7 | 82.3 | 85.9 | 88.5 | 80.9 | |
| BARTScore2023.07 | 74.3 | 62.6 | 91.7 | 82.3 | 85.9 | 88.5 | 80.9 | |
| SummaC-ZSType=NLI2023.05 | 73.6 | 58 | 87.5 | 83.7 | 85.8 | 85.3 | 79 | |
| BLEUType=Similarity Matching2023.05 | 71.8 | 55.8 | 86.9 | 75 | 83.8 | 84.5 | 76.3 | |
| ROUGE-2Type=Similarity Matching2023.05 | 70.5 | 65.9 | 83.7 | 76 | 87.2 | 85.3 | 78.1 | |
| ROUGE-LType=Similarity Matching2023.05 | 70.2 | 62.9 | 81.9 | 76.3 | 87.3 | 85.3 | 77.3 | |
| ROUGE-1Type=Similarity Matching2023.05 | 69.7 | 64.5 | 82.5 | 75.8 | 87.2 | 85 | 77.4 | |
| CTCType=Misc2023.05 | 69.1 | 61.7 | 82.1 | 77.6 | 78.4 | 80.5 | 74.9 | |
| SummaC-CONVType=NLI2023.05 | 67.2 | 70.3 | 81.8 | 92.3 | 86.1 | 88.5 | 81 | |
| FactCCType=Misc2023.05 | 64.9 | 55.1 | 78.5 | 72.7 | 71.8 | 69.8 | 68.8 | |
| FactCCType=Misc2023.05 | 64.9 | 55.1 | 78.5 | 72.7 | 71.8 | 69.8 | 68.8 | |
| BLEUType=Similarity Matching2023.05 | 64.2 | 55.2 | 78.3 | 67 | 77.6 | 79.3 | 70.3 | |
| BERTScoreType=Similarity Matching, Backbone=DeBERTa-XLarge-MNLI2023.05 | 63.1 | 49 | 85.3 | 70.9 | 79.6 | 84.9 | 72.1 | |
| BERTScore2023.07 | 63.1 | 49 | 85.3 | 70.9 | 79.6 | 84.9 | 72.1 | |
| SummaC-ZSType=NLI2023.05 | 62.6 | 57.8 | 81 | 82.8 | 77.8 | 78.1 | 73.4 | |
| ROUGE-LType=Similarity Matching2023.05 | 61.5 | 57.4 | 74 | 67.7 | 79.7 | 78.8 | 69.8 | |
| ROUGE-2Type=Similarity Matching2023.05 | 61.2 | 62.2 | 75.1 | 67.8 | 78.8 | 78.8 | 70.7 | |
| ROUGE-1Type=Similarity Matching2023.05 | 61.1 | 62.4 | 74.4 | 68 | 80 | 79.1 | 70.8 | |
| BLEURTType=Regression, Model=BLEURT-202023.05 | 60.8 | 64.7 | 76.7 | 59.7 | 71.1 | 82.5 | 69.2 | |
| BLEURT2023.07 | 60.8 | 64.7 | 76.7 | 59.7 | 71.1 | 82.5 | 69.2 | |
| QuestEvalType=QA2023.05 | 60.4 | 63.6 | 77 | 74.2 | 74.3 | 85.8 | 72.5 | |
| SummaC-CONVType=NLI2023.05 | 59.8 | 66.4 | 73.7 | 89.2 | 79.8 | 81 | 75 | |
| BLEURTType=Regression2023.05 | 57.7 | 58.7 | 69 | 56.2 | 63.7 | 74.9 | 63.4 | |
| BARTScoreType=Misc2023.05 | 56.9 | 58.7 | 84.6 | 73.3 | 79.6 | 78.3 | 71.9 | |
| SimCSEType=Similarity Matching, Model=sup-simcse-roberta-large2023.05 | 56.2 | 62.2 | 75.2 | 59 | 77.2 | 74.8 | 67.4 | |
| SimCSEType=Similarity Matching2023.05 | 54.4 | 57.3 | 68.9 | 57.3 | 71.3 | 68.5 | 62.9 | |
| BLANCType=Misc2023.05 | 54.1 | 53.5 | 74.7 | 56.4 | 68.6 | 83.4 | 65.1 | |
| FEQAType=QA2023.05 | 53.7 | 47.6 | 54.3 | 47.9 | 48.8 | 37.2 | 48.3 | |
| QuestEvalType=QA2023.05 | 53.1 | 57.6 | 69.3 | 66.8 | 69.8 | 77.7 | 65.7 | |
| BERTScoreType=Similarity Matching2023.05 | 52.7 | 49 | 76.9 | 65.3 | 72.7 | 78.5 | 65.8 | |
| DAEType=NLI2023.05 | 52.4 | 76.7 | 72.8 | 54.2 | 66.1 | 78.9 | 66.8 | |
| DAEType=NLI2023.05 | 52.4 | 76.7 | 72.8 | 54.2 | 66.1 | 78.9 | 66.8 | |
| FEQAType=QA2023.05 | 51.9 | 49.5 | 53.7 | 46.6 | 51.4 | 41.4 | 49.1 | |
| NER-OverlapType=Similarity Matching2023.05 | 51.1 | 64.9 | 72.1 | 49.8 | 56.6 | 68.1 | 60.4 | |
| NER-OverlapType=Similarity Matching2023.05 | 51.1 | 64.9 | 72.1 | 49.8 | 56.6 | 68.1 | 60.4 | |
| QAFactEvalType=QA2023.05 | 50.6 | 61.2 | 60.2 | 73.8 | 54.9 | 74.9 | 62.6 | |
| BLANCType=Misc2023.05 | 49.8 | 52 | 66.3 | 55.7 | 58.3 | 78.4 | 60.1 | |
| MNLIType=NLI2023.05 | 46 | 48.7 | 46.3 | 52.2 | 50.7 | 55.2 | 49.8 | |
| MNLIType=NLI, Model=RoBERTa-large2023.05 | 44.9 | 46.6 | 45 | 48.3 | 43.5 | 59.3 | 47.9 |