Text Summarization on NYT (test)
49.18R1 ScoreORACLE
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| ORACLEevaluation_protocol=limited-length ROUGE Recall, type=upper bound2019.08 | 49.18 | 33.24 | 46.02 | |
| BERTSUMEXTABSevaluation_protocol=limited-length ROUGE Recall, type=abstractive, backbone=BERT2019.08 | 49.02 | 31.02 | 45.55 | |
| BERTSUMABSevaluation_protocol=limited-length ROUGE Recall, type=abstractive, backbone=BERT2019.08 | 48.92 | 30.84 | 45.41 | |
| BERTSUMEXTevaluation_protocol=limited-length ROUGE Recall, type=extractive, backbone=BERT2019.08 | 46.66 | 26.35 | 42.62 | |
| PTGEN + COVevaluation_protocol=limited-length ROUGE Recall, type=abstractive2019.08 | 43.71 | 26.4 | — | |
| DRMevaluation_protocol=limited-length ROUGE Recall, type=abstractive2019.08 | 42.94 | 26.02 | — | |
| PTGENevaluation_protocol=limited-length ROUGE Recall, type=abstractive2019.08 | 42.47 | 25.61 | — | |
| SUMOevaluation_protocol=limited-length ROUGE Recall, type=extractive2019.08 | 42.3 | 22.7 | 38.6 | |
| COMPRESSevaluation_protocol=limited-length ROUGE Recall, type=extractive2019.08 | 42.2 | 24.9 | — | |
| TransformerEXTevaluation_protocol=limited-length ROUGE Recall, type=extractive, pre-trained=false2019.08 | 41.95 | 22.68 | 38.51 | |
| LEAD-3evaluation_protocol=limited-length ROUGE Recall, type=baseline2019.08 | 39.58 | 20.11 | 35.78 | |
| TransformerABSevaluation_protocol=limited-length ROUGE Recall, type=abstractive2019.08 | 35.75 | 17.23 | 31.41 | |
| DIONYSUSModel Size=Large, Evaluation Protocol=Zero-shot, Strategy=Better ROUGE2022.12 | 24.1 | 4 | 15.4 | |
| PEGASUSModel Size=Large, Evaluation Protocol=Zero-shot2022.12 | 23.7 | 3.2 | 13.2 | |
| DIONYSUSModel Size=Large, Evaluation Protocol=Zero-shot, Strategy=All G2022.12 | 21.7 | 3.7 | 14.8 | |
| DIONYSUSModel Size=Large, Evaluation Protocol=Zero-shot, Strategy=All P2022.12 | 19.8 | 2.7 | 12.9 | |
| GSG+Evaluation Protocol=Zero-shot2022.12 | 17.1 | 2.4 | 12.9 | |
| T5v1.1Model Size=Large, Evaluation Protocol=Zero-shot2022.12 | 11.6 | 1.4 | 8.7 |