Query-focused Summarization on QMSum (test)
38.67ROUGE-1IDEALLoRA
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| IDEALLoRALocal context size=8K2024.07 | 38.67 | 14.42 | 26.28 | — | 34.24 | 87.29 | |
| Qontsum2024.07 | 38.42 | 13.5 | — | — | 34.03 | 87.72 | |
| BART-Large SegEncPre-training objective=SOCRATIC, Pre-training instances=Full Books32022.12 | 38.06 | 13.74 | 33.51 | 87.63 | — | — | |
| SegEnc+Socratic Pret.2024.07 | 38.06 | 13.74 | — | — | 33.51 | 87.63 | |
| BART-LS2022.12 | 37.9 | 12.1 | 33.1 | — | — | — | |
| BART-Large SegEncPre-Finetuning=WikiSum2022.12 | 37.8 | 13.43 | 33.38 | — | — | — | |
| BART-Large SegEncPre-training objective=SOCRATIC, Pre-training instances=1M2022.12 | 37.46 | 13.32 | 32.79 | 87.54 | — | — | |
| BART-Large SegEncPre-training objective=PEGASUS (GSG), Pre-training instances=Full Books32022.12 | 37.29 | 13.3 | 32.7 | 87.48 | — | — | |
| IDEALPAdapterLocal context size=8K2024.07 | 37.27 | 13.9 | 26.32 | — | 32.73 | 87.19 | |
| BART-Large SegEnc2022.12 | 37.05 | 13.04 | 32.62 | 87.44 | — | — | |
| SegEnc2024.07 | 37.05 | 13.03 | — | — | 32.62 | 87.44 | |
| BART-Large SegEncPre-training objective=Continued Pre-training, Pre-training instances=1M2022.12 | 36.64 | 12.44 | 31.94 | 86.94 | — | — | |
| LoraLocal context size=8K2024.07 | 36.4 | 12.1 | 23.98 | — | 31.87 | 86.66 | |
| PAdapterLocal context size=8K2024.07 | 36.03 | 12.61 | 24.64 | — | 31.74 | 86.96 | |
| HMNet+Locator2024.07 | 32.29 | 8.67 | — | — | 28.17 | — | |
| Bart-largeLocal context size=1K2024.07 | 31.76 | 7.76 | 20.02 | — | 27.52 | 85.22 | |
| IDEALPromptLocal context size=8K2024.07 | 31.41 | 10.6 | 22.27 | — | 27.19 | 86.08 | |
| Bart+UnlimiformerLocal context size=1K2024.07 | 30.9 | 8 | 19.9 | — | — | — | |
| PromptLocal context size=8K2024.07 | 30.52 | 9.77 | 21.5 | — | 26.33 | 85.89 | |
| LED-base-OASumLocal context size=4K2024.07 | 30.3 | 7.56 | — | — | 26.67 | — | |
| GPT-4OLocal context size=16K2024.07 | 30.3 | 8.13 | 18.34 | — | 26.8 | 84.4 | |
| ChatGPT2024.07 | 28.34 | 8.74 | 17.81 | — | 18.81 | — | |
| Llama3.1-8BLocal context size=16K2024.07 | 21.34 | 6.26 | 14.39 | — | 19.03 | 82.69 |