Faithfulness Measurement on tldr_news
79.4BLEURandom
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| RandomModel=LlaMA-2 (7B-Chat)2024.05 | 79.4 | 74.2 | 74.1 | 74.1 | 92.3 | 94.4 | 0.037 | |
| RandomModel=Vicuna (7B)2024.05 | 78.1 | 73.6 | 74.6 | 73.7 | 92.1 | 89.1 | 0.091 | |
| AttentionModel=LlaMA-2 (7B-Chat)2024.05 | 76.7 | 70.3 | 71 | 70.6 | 90 | 89.9 | 0.077 | |
| CaptumModel=LlaMA-2 (7B-Chat)2024.05 | 75.9 | 70.1 | 70.3 | 70.1 | 90 | 91 | 0.069 | |
| Last-AttentionModel=LlaMA-2 (7B-Chat)2024.05 | 74.7 | 68.3 | 68.5 | 68.3 | 87.6 | 86.9 | 0.108 | |
| Last-AttentionModel=Vicuna (7B)2024.05 | 74.6 | 70.7 | 71.8 | 70.8 | 89.8 | 87.6 | 0.114 | |
| Integrated-GradientModel=LlaMA-2 (7B-Chat)2024.05 | 71.3 | 64.1 | 64.2 | 64 | 86.6 | 81.7 | 0.149 | |
| Integrated-GradientModel=Vicuna (7B)2024.05 | 70.5 | 66.9 | 67.2 | 66.6 | 87.4 | 82.1 | 0.183 | |
| JoPAModel=LlaMA-2 (7B-Chat)2024.05 | 69.2 | 61.9 | 61 | 61.2 | 84.1 | 60.4 | 0.394 | |
| AttentionModel=Vicuna (7B)2024.05 | 62.1 | 51.4 | 51.5 | 50.9 | 81.7 | 71.4 | 0.294 | |
| CaptumModel=Vicuna (7B)2024.05 | 55.6 | 47 | 46.1 | 45.9 | 77.5 | 37.6 | 0.829 | |
| JoPAModel=Vicuna (7B)2024.05 | 53.6 | 45.6 | 45.4 | 44.8 | 77.2 | 31.7 | 1.006 |