Multi-document summarization on WCEP (test)
46.29R-1SimCAS
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| SimCASBackbone=BART-large2023.08 | 46.29 | 24.45 | 38.61 | — | 71.38 | |
| PRIMERATraining=Fully Supervised2021.10 | 46.08 | 25.21 | 37.86 | — | — | |
| PRIMERA2023.08 | 46.08 | 25.21 | 37.86 | — | — | |
| SimCASBackbone=BART-base2023.08 | 45.68 | 22.8 | 37.71 | — | 70.59 | |
| ROUGE-L + RELAXoptimization=RELAX, reward=ROUGE-L2022.03 | 41.11 | 19.46 | 33.13 | 30.57 | — | |
| LED+RELAX2023.08 | 41.11 | 19.46 | 33.13 | — | — | |
| ROUGE-L + Coverage + RELAXoptimization=RELAX, reward=ROUGE-L, coverage_term=true, beta=12022.03 | 40.78 | 19.14 | 32.37 | 32.21 | — | |
| ROUGE-L + Coverage + REINFORCEoptimization=REINFORCE, reward=ROUGE-L, coverage_term=true, beta=12022.03 | 40.68 | 18.8 | 32.71 | 30.28 | — | |
| ROUGE-L + REINFORCEoptimization=REINFORCE, reward=ROUGE-L2022.03 | 40.25 | 18.18 | 31.58 | 30.91 | — | |
| BART-LED (Baseline)architecture=BART Longformer, objective=NLL2022.03 | 39.79 | 18.94 | 32.1 | 29.04 | — | |
| LEDBackbone=large2023.08 | 39.79 | 18.94 | 32.1 | — | — | |
| BARTBackbone=BART-large2023.08 | 37.66 | 15.98 | 31.01 | — | 62.32 | |
| BARTBackbone=BART-base2023.08 | 36.12 | 13.77 | 29.98 | — | 60.84 | |
| BART-WCEP-DynE-52022.03 | 35.4 | 15.1 | 25.6 | — | — | |
| DynETraining=Fully Supervised2021.10 | 35.4 | 15.1 | 25.6 | — | — | |
| DynE2023.08 | 35.4 | 15.1 | 25.6 | — | — | |
| TSR2022.03 | 35.3 | 13.7 | 25.7 | — | — | |
| BERTReg2022.03 | 35 | 13.5 | 25.5 | — | — | |
| BERTREGTraining=Fully Supervised2021.10 | 35 | 13.5 | 25.5 | — | — | |
| BERTREG2023.08 | 35 | 13.5 | 25.5 | — | — | |
| Submodular+ABS2022.03 | 34.4 | 13.1 | 25 | — | — | |
| SUBMODULAR+ABS2023.08 | 34.4 | 13.1 | 25 | — | — | |
| PEGASUSzero-shot=true, source=our run, output length limit=502021.10 | 33.2 | 12.7 | 23.8 | — | — | |
| SUBMODULAR+ABSTraining=Fully Supervised2021.10 | 30.6 | 10.1 | 21.4 | — | — | |
| PRIMERAzero-shot=true, output length limit=502021.10 | 28 | 10.3 | 20.9 | — | — | |
| BARTzero-shot=true, source=our run, output length limit=502021.10 | 20.2 | 5.7 | 15.3 | — | — | |
| LEDzero-shot=true, source=our run, output length limit=502021.10 | 18.8 | 5.4 | 14.7 | — | — |