Opinion Summarization Evaluation on SUMMEVAL-OP Round-I
0.63Relevance (RE) Spearman RhoOP-MISTRAL
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OP-MISTRALBackbone=Mistral-7B2024.02 | 0.63 | — | — | — | — | 0.55 | 0.59 | 0.45 | 0.57 | 0.46 | 0.8 | 0.67 | 0.6 | 0.49 | |
| OP-I-MISTRALBackbone=Mistral-7B2024.02 | 0.59 | 0.54 | 0.45 | 0.58 | 0.47 | 0.47 | 0.63 | 0.51 | 0.82 | 0.7 | 0.73 | 0.61 | 0.71 | 0.58 | |
| OP-I-GPT-3.5Backbone=GPT-3.52024.02 | 0.56 | — | — | 0.51 | 0.69 | 0.71 | 0.59 | 0.8 | 0.68 | 0.73 | 0.61 | 0.74 | 0.61 | 0.71 | |
| G-EVAL 3.5Backbone=GPT-3.52024.02 | 0.49 | 0.45 | 0.36 | 0.28 | 0.52 | 0.68 | 0.56 | 0.7 | 0.58 | 0.79 | 0.67 | 0.73 | 0.61 | 0.75 | |
| G-EVAL-MISTRALBackbone=Mistral-7B2024.02 | 0.38 | 0.5 | 0.43 | 0.54 | 0.45 | 0.32 | 0.58 | 0.47 | 0.54 | 0.44 | 0.61 | 0.49 | 0.55 | 0.46 | |
| SUMMAC2024.02 | 0.3 | 0.23 | 0.2 | 0.18 | 0.14 | 0.25 | 0.25 | 0.21 | 0.24 | 0.19 | 0.25 | 0.2 | 0.26 | 0.21 | |
| UNIEVAL2024.02 | 0.25 | 0.59 | 0.52 | 0.42 | 0.33 | 0.17 | 0.14 | — | — | — | — | — | — | — | |
| ROUGE-22024.02 | -0.17 | -0.23 | -0.18 | -0.14 | -0.1 | -0.12 | -0.21 | -0.16 | -0.26 | -0.19 | -0.24 | -0.18 | -0.14 | -0.09 | |
| BERTSCORE2024.02 | -0.29 | -0.32 | -0.27 | -0.28 | -0.22 | -0.22 | -0.34 | -0.26 | -0.51 | -0.43 | -0.41 | -0.33 | -0.37 | -0.28 | |
| BARTSCORE2024.02 | -0.29 | -0.19 | -0.15 | -0.19 | -0.14 | -0.22 | -0.33 | -0.25 | -0.45 | -0.35 | -0.37 | -0.28 | -0.36 | -0.27 | |
| ROUGE-12024.02 | -0.31 | -0.36 | -0.28 | -0.3 | -0.24 | -0.23 | -0.35 | -0.26 | -0.44 | -0.32 | -0.38 | -0.29 | -0.3 | -0.23 | |
| ROUGE-L2024.02 | -0.34 | -0.39 | -0.32 | -0.3 | -0.23 | -0.25 | -0.4 | -0.3 | -0.51 | -0.37 | -0.45 | -0.33 | -0.38 | -0.27 |