Query-focused Summarization on Squality (test)
46.6ROUGE-1Human
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Human2022.12 | 46.6 | 12.5 | 22.7 | — | — | — | |
| BART-Large SegEncPre-training objective=SOCRATIC2022.12 | 46.31 | 14.8 | 22.76 | 86.04 | — | — | |
| SegEnc+Socratic Pret.2024.07 | 46.31 | 14.8 | 22.76 | — | — | 86.04 | |
| BART-Large SegEncPre-training objective=PEGASUS (GSG)2022.12 | 45.78 | 14.43 | 22.9 | 85.94 | — | — | |
| Qontsum2024.07 | 45.76 | 14.27 | 24.14 | — | — | 86.07 | |
| BART-Large SegEnc2022.12 | 45.68 | 14.51 | 22.47 | 85.86 | — | — | |
| SegEnc2024.07 | 45.68 | 14.51 | 22.47 | — | — | 85.86 | |
| LoraLocal context size=8K2024.07 | 44.47 | 13.23 | 24.32 | — | 41.46 | 86.63 | |
| IDEALPAdapterLocal context size=8K2024.07 | 44.37 | 13.43 | 24.76 | — | 41.47 | 86.66 | |
| IDEALLoRALocal context size=8K2024.07 | 43.87 | 13.86 | 25.54 | — | 40.99 | 86.86 | |
| PAdapterLocal context size=8K2024.07 | 43.86 | 13.13 | 24.21 | — | 40.83 | 86.55 | |
| BARTRetrieval augmentation=DPR2022.12 | 41.5 | 11.4 | 21 | — | — | — | |
| Bart-Large+DPRLocal context size=1K2024.07 | 41.5 | 11.4 | 21 | — | — | 85.5 | |
| GPT-4OLocal context size=8K2024.07 | 41.01 | 9.84 | 21.02 | — | 36.99 | 85.39 | |
| BART2022.12 | 40.2 | 10.4 | 20.8 | — | — | — | |
| Bart-largeLocal context size=1K2024.07 | 38.58 | 9.81 | 20.97 | — | 36.11 | 84.81 | |
| PEGASUS2022.12 | 38.2 | 9 | 20.2 | — | — | — | |
| LED-base-OASumLocal context size=4K2024.07 | 37.6 | 8.81 | — | — | 35.14 | — | |
| IDEALPromptLocal context size=8K2024.07 | 37.4 | 11.23 | 23.93 | — | 34.44 | 85.56 | |
| ChatGPT2024.07 | 37.02 | 8.19 | 18.45 | — | 22.56 | — | |
| Llama3.1-8BLocal context size=8K2024.07 | 36.84 | 9.34 | 20.14 | — | 34.17 | 84.39 | |
| PromptLocal context size=8K2024.07 | 36.81 | 10.72 | 23.69 | — | 33.26 | 85.23 | |
| LED2022.12 | 27.7 | 5.9 | 17.7 | — | — | — |