Abstractive Dialogue Summarization on SAMSum (test)
52.7ROUGE-Lours
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| oursStudent Model Architecture=T5-Large, Teacher Model=T5-XL2024.06 | 52.7 | — | — | — | |
| T5-XL (teacher)Model Architecture=T5-XL2024.06 | 52.5 | — | — | — | |
| DistiLLMStudent Model Architecture=T5-Large, Teacher Model=T5-XL2024.06 | 52.1 | — | — | — | |
| GKDStudent Model Architecture=T5-Large, Teacher Model=T5-XL2024.06 | 51.7 | — | — | — | |
| ImitKDStudent Model Architecture=T5-Large, Teacher Model=T5-XL2024.06 | 51.2 | — | — | — | |
| CODS2022.09 | 50.79 | 52.65 | 27.84 | 66.55 | |
| oursStudent Model Architecture=T5-Base, Teacher Model=T5-XL2024.06 | 50.4 | — | — | — | |
| SFT (student)Student Model Architecture=T5-Large, Teacher Model=T5-XL2024.06 | 49.8 | — | — | — | |
| SICK++Commonsense knowledge source=PARA-COMET2022.09 | 49.5 | 53.73 | 28.81 | 71.92 | |
| DistiLLMStudent Model Architecture=T5-Base, Teacher Model=T5-XL2024.06 | 49.4 | — | — | — | |
| PEGASUSRe-implementation status=obtained from Wu et al. (2021)2022.09 | 49.32 | 50.5 | 27.23 | — | |
| SeqKDStudent Model Architecture=T5-Large, Teacher Model=T5-XL2024.06 | 49.3 | — | — | — | |
| SICKCommonsense knowledge source=PARA-COMET2022.09 | 49.12 | 53.39 | 28.42 | 71.83 | |
| SICK++Commonsense knowledge source=COMET2022.09 | 48.9 | 53.24 | 28.1 | 71.71 | |
| BART-xsumRe-implementation status=re-implemented2022.09 | 48.75 | 52.5 | 27.67 | 68.16 | |
| BART-largeRe-implementation status=re-implemented2022.09 | 48.72 | 51.74 | 26.46 | 53.87 | |
| ImitKDStudent Model Architecture=T5-Base, Teacher Model=T5-XL2024.06 | 48.6 | — | — | — | |
| SICKCommonsense knowledge source=COMET2022.09 | 48.49 | 53.04 | 27.6 | 71.61 | |
| GKDStudent Model Architecture=T5-Base, Teacher Model=T5-XL2024.06 | 48.2 | — | — | — | |
| S-BART2022.09 | 48.08 | 50.7 | 25.5 | 70.07 | |
| SeqKDStudent Model Architecture=T5-Base, Teacher Model=T5-XL2024.06 | 47.7 | — | — | — | |
| KDStudent Model Architecture=T5-Large, Teacher Model=T5-XL2024.06 | 47.4 | — | — | — | |
| SFT (student)Student Model Architecture=T5-Base, Teacher Model=T5-XL2024.06 | 47.3 | — | — | — | |
| UniLMRe-implementation status=obtained from Wu et al. (2021)2022.09 | 46.67 | 49.88 | 24.23 | — | |
| KDStudent Model Architecture=T5-Base, Teacher Model=T5-XL2024.06 | 46.5 | — | — | — | |
| Z-Code++ Largetraining_examples=1000, pretraining=Two-phase2022.08 | 43.9 | 52.2 | 28.1 | — | |
| oursStudent Model Architecture=T5-Small, Teacher Model=T5-XL2024.06 | 43.7 | — | — | — | |
| DistiLLMStudent Model Architecture=T5-Small, Teacher Model=T5-XL2024.06 | 42.6 | — | — | — | |
| Z-Code++ Large (Phase 1)training_examples=1000, pretraining=Phase 1 only2022.08 | 42.3 | 50.9 | 26.3 | — | |
| GKDStudent Model Architecture=T5-Small, Teacher Model=T5-XL2024.06 | 42.1 | — | — | — | |
| ImitKDStudent Model Architecture=T5-Small, Teacher Model=T5-XL2024.06 | 41.8 | — | — | — | |
| PEGASUS-Largetraining_examples=10002022.08 | 40.6 | 49.3 | 24.4 | — | |
| SFT (student)Student Model Architecture=T5-Small, Teacher Model=T5-XL2024.06 | 40.6 | — | — | — | |
| SeqKDStudent Model Architecture=T5-Small, Teacher Model=T5-XL2024.06 | 39.7 | — | — | — | |
| D-HGN2022.09 | 39.57 | 43.03 | 18.07 | 64.2 | |
| TransformerRe-implementation status=obtained from Gliwa et al. (2019)2022.09 | 39.27 | 42.37 | 18.44 | — | |
| KDStudent Model Architecture=T5-Small, Teacher Model=T5-XL2024.06 | 39.2 | — | — | — | |
| Z-Code++ Largetraining_examples=100, pretraining=Two-phase2022.08 | 38.7 | 47.6 | 22.3 | — | |
| DialoGPTRe-implementation status=re-implemented2022.09 | 38.42 | 39.77 | 16.58 | — | |
| DynamicConvRe-implementation status=obtained from Gliwa et al. (2019)2022.09 | 37.27 | 41.07 | 17.11 | — | |
| Z-Code++ Large (Phase 1)training_examples=100, pretraining=Phase 1 only2022.08 | 36.7 | 44.7 | 20.2 | — | |
| PEGASUS-Largetraining_examples=1002022.08 | 36.1 | 45 | 19.8 | — | |
| PointerGeneratorRe-implementation status=obtained from Gliwa et al. (2019)2022.09 | 34.36 | 32.27 | 14.42 | — | |
| Z-Code++ Largetraining_examples=10, pretraining=Two-phase2022.08 | 33.7 | 40.27 | 17.4 | — | |
| T5-Largetraining_examples=10002022.08 | 32.8 | 41.4 | 17.8 | — | |
| PEGASUS-Largetraining_examples=102022.08 | 28.1 | 37 | 11.7 | — | |
| T5-Largetraining_examples=1002022.08 | 23.5 | 29.6 | 10.4 | — | |
| PEGASUS-Largetraining_examples=0, zero-shot=true2022.08 | 20.5 | 26.3 | 6.4 | — | |
| Z-Code++ Largetraining_examples=0, zero-shot=true, pretraining=Two-phase2022.08 | 20.5 | 26.5 | 7.9 | — | |
| T5-Largetraining_examples=102022.08 | 12 | 14 | 4 | — | |
| Z-Code++ Large (Phase 1)training_examples=10, pretraining=Phase 1 only2022.08 | 11 | 13.6 | 2.6 | — | |
| T5-Largetraining_examples=0, zero-shot=true2022.08 | 8.2 | 9.4 | 1.3 | — | |
| Z-Code++ Large (Phase 1)training_examples=0, zero-shot=true, pretraining=Phase 1 only2022.08 | 5.4 | 6 | 0.1 | — | |
| AugBackbone=BART-large2023.05 | — | — | 28.29 | 75.26 | |
| EmbBackbone=BART-large2023.05 | — | — | 28.12 | 75.14 | |
| InsBackbone=BART-large2023.05 | — | — | 28.97 | 75.63 | |
| VanillaBackbone=BART-large2023.05 | — | — | 28.12 | 75.09 |