Summarization Evaluation on SummEval
57CoherenceSMART-L
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| SMART-LReference Dependency=Reference Dependent2024.02 | 57 | 57 | 61 | 73 | — | |
| SMART-LReference-dependence=Reference-dependent2022.12 | 56.7 | 56.7 | 61.1 | 73.3 | 61.9 | |
| BARTScore+CNNReference-dependence=Reference-free, Training Dataset=CNN/DM2022.12 | 55 | 31.7 | 59.4 | 58.3 | 51.1 | |
| BLEURTReference-dependence=Reference-dependent2022.12 | 53.3 | 20 | 41 | 46.7 | 40.3 | |
| RISE Multi-NewsReference-dependence=Reference-free, Model Size=Large, Training Dataset=Multi-News2022.12 | 53.3 | 73.3 | 71.1 | 70 | 66.9 | |
| RISE SamSUMReference-dependence=Reference-free, Model Size=Large, Training Dataset=SamSUM2022.12 | 53.3 | 70 | 67.8 | 70 | 65.3 | |
| RISE CNNReference-dependence=Reference-free, Model Size=Large, Training Dataset=CNN/DM2022.12 | 53.3 | 73.3 | 74.5 | 70 | 67.8 | |
| BLEURTReference Dependency=Reference Dependent2024.02 | 53 | 20 | 41 | 47 | — | |
| RISE extMulti-NewsReference Dependency=Reference Free2024.02 | 53 | 73 | 71 | 70 | — | |
| RISE SamSUMReference Dependency=Reference Free2024.02 | 53 | 70 | 68 | 70 | — | |
| RISE CNNReference Dependency=Reference Free2024.02 | 53 | 73 | 75 | 70 | — | |
| SMART-1Reference-dependence=Reference-dependent2022.12 | 43.3 | 66.7 | 64.4 | 66.7 | 60.3 | |
| SMART-1Reference Dependency=Reference Dependent2024.02 | 43 | 67 | 64 | 67 | — | |
| SMART-2Reference Dependency=Reference Dependent2024.02 | 42 | 75 | 63 | 58 | — | |
| SMART-2Reference-dependence=Reference-dependent2022.12 | 41.7 | 75 | 62.8 | 58.3 | 59.4 | |
| ROUGE-1Reference-dependence=Reference-dependent2022.12 | 35 | 55 | 52.7 | 58.3 | 50.3 | |
| CHRFReference-dependence=Reference-dependent2022.12 | 35 | 61.7 | 56.1 | 55 | 51.9 | |
| BARTScoreReference-dependence=Reference-free2022.12 | 35 | 61.7 | 49.4 | 45 | 47.8 | |
| ROUGE-1Reference Dependency=Reference Dependent2024.02 | 35 | 55 | 53 | 58 | — | |
| CHRFReference Dependency=Reference Dependent2024.02 | 35 | 63 | 56 | 55 | — | |
| BARTScoreReference Dependency=Reference Free2024.02 | 35 | 62 | 49 | 45 | — | |
| BERTScoreReference-dependence=Reference-dependent2022.12 | 33.3 | -3 | 14.2 | 20 | 16.1 | |
| BERTScoreReference Dependency=Reference Dependent2024.02 | 33 | -3 | 14 | 20 | — | |
| BARTScore+CNNReference Dependency=Reference Free2024.02 | 32 | 59 | 45 | 58 | — | |
| SMSReference Dependency=Reference Dependent2024.02 | 27 | 60 | 36 | 40 | — | |
| SMSReference-dependence=Reference-dependent2022.12 | 26.7 | 60 | 36 | 40 | 40.7 | |
| T5-ANLIReference-dependence=Reference-free2022.12 | 25 | 58.3 | 54.4 | 51.7 | 47.3 | |
| Q2Reference-dependence=Reference-free2022.12 | 25 | 75 | 57.7 | 45 | 50.7 | |
| T5-ANLIReference Dependency=Reference Free2024.02 | 25 | 58 | 54 | 52 | — | |
| Q2Reference Dependency=Reference Free2024.02 | 25 | 75 | 58 | 45 | — | |
| ROUGE-2Reference-dependence=Reference-dependent2022.12 | 23.3 | 60 | 49.4 | 43.3 | 44 | |
| PRISMReference-dependence=Reference-free2022.12 | 23.3 | 60 | 36 | 36.7 | 39 | |
| ROUGE-2Reference Dependency=Reference Dependent2024.02 | 23 | 60 | 49 | 43 | — | |
| MoverScoreReference Dependency=Reference Dependent2024.02 | 23 | -5 | 26 | 35 | — | |
| PRISMReference Dependency=Reference Free2024.02 | 23 | 60 | 36 | 37 | — | |
| COSMIC (I(T; S))Reference Dependency=Reference Free2024.02 | 23 | 53 | 47 | 54 | — | |
| BLEUReference Dependency=Reference Dependent2024.02 | 22 | 5 | 33 | 38 | — | |
| BLEUReference-dependence=Reference-dependent2022.12 | 21.7 | 5 | 32.6 | 38.3 | 24.4 | |
| MoverScoreReference-dependence=Reference-dependent2022.12 | 21.7 | -5 | 25.9 | 35 | 19.4 | |
| ROUGE-LReference Dependency=Reference Dependent2024.02 | 12 | 12 | 26 | 35 | — | |
| ROUGE-LReference-dependence=Reference-dependent2022.12 | 11.7 | 11.7 | 25.9 | 35 | 21.1 |