Persona-based Summarization on WebMD (test)
98.2RelevanceGPT-4
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| GPT-4role=Gold Standard / Zero-shot2024.06 | 98.2 | 96.3 | 98.6 | 98.5 | 99.7 | |
| L-F-13bbackbone=Llama2-13b, mode=fine-tuned2024.06 | 93.5 | 90.1 | 91.7 | 88.5 | 99.1 | |
| L-F-7bbackbone=Llama2-7b, mode=fine-tuned2024.06 | 75.8 | 58.7 | 85.8 | 63.8 | 58.6 | |
| T5-FTtype=fine-tuned2024.06 | 72.4 | 70.1 | 84.9 | 67.6 | 78.1 | |
| FT5-FTtype=fine-tuned2024.06 | 72.2 | 70.2 | 88 | 68.3 | 80.3 | |
| LED-Lsize=large2024.06 | 69.1 | 56.2 | 82.3 | 59.3 | 56.6 | |
| BART2024.06 | 65.4 | 42.6 | 84.8 | 49.7 | 25.8 | |
| Falcon2024.06 | 56.3 | 45.4 | 82 | 50.6 | 46.8 | |
| L-V-70bbackbone=Llama2-70b, mode=vanilla/zero-shot2024.06 | 49.7 | 45.1 | 78 | 46.4 | 47.8 | |
| Pegasus2024.06 | 47.7 | 33 | 74 | 36.1 | 11.5 | |
| LED-Bsize=base2024.06 | 36.1 | 19.2 | 79.3 | 31.3 | 17.6 | |
| L-V-13bbackbone=Llama2-13b, mode=vanilla/zero-shot2024.06 | 32.1 | 29.1 | 73.1 | 28.5 | 23.4 | |
| L-V-7bbackbone=Llama2-7b, mode=vanilla/zero-shot2024.06 | 19.1 | 18.4 | 41.3 | 16.6 | 15.2 |