Summarization on CNN/DailyMail (test)
48ROUGE-LFreeTxt-Vi (fine-tuned Qwen2.5)
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FreeTxt-Vi (fine-tuned Qwen2.5)Speed (tok/s)=118, Evaluation protocol=fine-tuned2026.03 | 48 | — | — | — | — | — | — | — | — | — | 52.4 | 27 | |
| Qwen2.5 (Base)Speed (tok/s)=1202026.03 | 46 | — | — | — | — | — | — | — | — | — | 50 | 25.2 | |
| Gemma-7BSpeed (tok/s)=1252026.03 | 45.6 | — | — | — | — | — | — | — | — | — | 49.1 | 24.5 | |
| LLaMA2-7BSpeed (tok/s)=1002026.03 | 44.9 | — | — | — | — | — | — | — | — | — | 48.3 | 23.9 | |
| Flan-T5-baseSpeed (tok/s)=1552026.03 | 44.5 | — | — | — | — | — | — | — | — | — | 47.8 | 24 | |
| mBART50Speed (tok/s)=1152026.03 | 43.7 | — | — | — | — | — | — | — | — | — | 46.4 | 23.1 | |
| T5-baseSpeed (tok/s)=1452026.03 | 43.2 | — | — | — | — | — | — | — | — | — | 45.9 | 22.6 | |
| BART-largeSpeed (tok/s)=1302026.03 | 41.9 | — | — | — | — | — | — | — | — | — | 44.7 | 21.3 | |
| HAT-BARTSource sequence length=10242021.04 | 41.52 | — | — | — | — | — | — | — | — | — | 44.48 | 21.31 | |
| Transformer-BARTSource sequence length=10242021.04 | 41.51 | — | — | — | — | — | — | — | — | — | 44.45 | 21.27 | |
| PEGASUS2021.04 | 41.3 | — | — | — | — | — | — | — | — | — | 44.16 | 21.56 | |
| BART2021.04 | 40.9 | — | — | — | — | — | — | — | — | — | 44.16 | 21.28 | |
| BigBirdSource sequence length=10242021.04 | 40.74 | — | — | — | — | — | — | — | — | — | 43.84 | 21.11 | |
| LoRABackbone=Qwen2.5-14B, Time=43:42:26, Memory=427602025.12 | 27.5 | — | — | — | — | — | — | — | — | — | 42.82 | 18.25 | |
| EABackbone=Qwen2.5-14B, Time=10:35:17, Memory=118342025.12 | 27.47 | — | — | — | — | — | — | — | — | — | 41.32 | 18.86 | |
| QLoRABackbone=Qwen2.5-14B, Time=50:33:47, Memory=206172025.12 | 27.35 | — | — | — | — | — | — | — | — | — | 42.19 | 18.03 | |
| FreezeBackbone=Qwen2.5-14B, Time=13:07:37, Memory=483712025.12 | 27.24 | — | — | — | — | — | — | — | — | — | 41.55 | 17.84 | |
| Qwen2.5-14BBackbone=Qwen2.5-14B, Configuration=Base/Zero-shot2025.12 | 26.14 | — | — | — | — | — | — | — | — | — | 36.82 | 13.27 | |
| EABackbone=GLM-4-9B, Time=8:33:06, Memory=118342025.12 | 26.1 | — | — | — | — | — | — | — | — | — | 40.17 | 15.91 | |
| LoRABackbone=GLM-4-9B, Time=24:32:13, Memory=327492025.12 | 26.06 | — | — | — | — | — | — | — | — | — | 40.25 | 16.34 | |
| QLoRABackbone=GLM-4-9B, Time=28:10:34, Memory=221252025.12 | 25.93 | — | — | — | — | — | — | — | — | — | 40.06 | 16.35 | |
| FreezeBackbone=GLM-4-9B, Time=9:06:55, Memory=364452025.12 | 25.77 | — | — | — | — | — | — | — | — | — | 39.86 | 15.68 | |
| GLM-4-9BBackbone=GLM-4-9B, Configuration=Base/Zero-shot2025.12 | 24.39 | — | — | — | — | — | — | — | — | — | 34.9 | 12.47 | |
| EoRAModel=LLaMA3-8B, Quantization Format=4-bit, Compensation Method=EoRA, Rank=1282024.10 | 18.12 | — | — | — | — | — | — | — | — | — | — | — | |
| ApiQModel=LLaMA3-8B, Quantization Format=4-bit, Compensation Method=ApiQ, Rank=1282024.10 | 18.04 | — | — | — | — | — | — | — | — | — | — | — | |
| ZeroQuant-V2Model=LLaMA3-8B, Quantization Format=4-bit, Compensation Method=ZeroQuant-V2, Rank=1282024.10 | 17.98 | — | — | — | — | — | — | — | — | — | — | — | |
| Act-SModel=LLaMA3-8B, Quantization Format=4-bit, Compensation Method=Act-S, Rank=1282024.10 | 17.86 | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA3-8BModel=LLaMA3-8B, Quantization Format=4-bit, Compensation Method=None2024.10 | 16.72 | — | — | — | — | — | — | — | — | — | — | — | |
| EoRAModel=LLaMA3-8B, Quantization Format=3-bit, Compensation Method=EoRA, Rank=1282024.10 | 14.63 | — | — | — | — | — | — | — | — | — | — | — | |
| ApiQModel=LLaMA3-8B, Quantization Format=3-bit, Compensation Method=ApiQ, Rank=1282024.10 | 13.57 | — | — | — | — | — | — | — | — | — | — | — | |
| Act-SModel=LLaMA3-8B, Quantization Format=3-bit, Compensation Method=Act-S, Rank=1282024.10 | 12.86 | — | — | — | — | — | — | — | — | — | — | — | |
| ZeroQuant-V2Model=LLaMA3-8B, Quantization Format=3-bit, Compensation Method=ZeroQuant-V2, Rank=1282024.10 | 9.7 | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA3-8BModel=LLaMA3-8B, Quantization Format=3-bit, Compensation Method=None2024.10 | 6.5 | — | — | — | — | — | — | — | — | — | — | — | |
| BART2019.10 | — | 44.16 | 21.28 | 40.9 | — | — | — | — | — | — | — | — | |
| BART2022.06 | — | — | — | — | — | 0.5056 | 88.47 | 57.54 | — | — | — | — | |
| BART BaseTraining Objective=Standard Cross-Entropy2023.09 | — | 42.4 | 19.3 | 29.3 | 39.6 | — | — | — | 16 | 36.4 | — | — | |
| BART Base + SDSTraining Objective=SDS (Self-supervised Objective for Summarization)2023.09 | — | 42.7 | 19.4 | 29.5 | 39.7 | — | — | — | 16.2 | 36.8 | — | — | |
| BERTSUMType=Extractive2019.08 | — | — | — | — | — | 58.9 | — | — | — | — | — | — | |
| BERTSUMABS2019.10 | — | 41.72 | 19.39 | 38.76 | — | — | — | — | — | — | — | — | |
| BERTSUMEXTABS2019.10 | — | 42.13 | 19.6 | 39.18 | — | — | — | — | — | — | — | — | |
| DODOcmpr.=10x2023.10 | — | 39.9 | 14.6 | 37 | — | — | — | — | — | — | — | — | |
| FULLcmpr.=1x, evaluation protocol=zero-shot2023.10 | — | 32.5 | 9.7 | 28.2 | — | — | — | — | — | — | — | — | |
| FULLcmpr.=1x, evaluation protocol=fine-tuning2023.10 | — | 37.7 | 15.6 | 35.3 | — | — | — | — | — | — | — | — | |
| Gold2018.09 | — | 33 | 30 | 24 | 13 | — | — | — | — | — | — | — | |
| GOLD2018.02 | — | 39 | 19 | 18 | 24 | — | — | — | — | — | — | — | |
| Hybrid MemNet2018.09 | — | 24 | 25 | 28 | 23 | — | — | — | — | — | — | — | |
| ITS2018.09 | — | 31 | 34 | 23 | 12 | — | — | — | — | — | — | — | |
| JGR-GComponent=Generator2022.06 | — | — | — | — | — | 0.5209 | 88.9 | 60.33 | — | — | — | — | |
| JGR-RComponent=Ranker2022.06 | — | — | — | — | — | 0.5218 | 88.96 | 61.59 | — | — | — | — | |
| LEAD2018.02 | — | 11 | 21 | 34 | 33 | 36.33 | — | — | — | — | — | — | |
| LEADType=Extractive2019.08 | — | — | — | — | — | 42.5 | — | — | — | — | — | — | |
| Lead-32018.09 | — | 12 | 11 | 25 | 52 | — | — | — | — | — | — | — | |
| Lead-32019.10 | — | 40.42 | 17.62 | 36.67 | — | — | — | — | — | — | — | — | |
| NEUSUMType=Extractive2019.08 | — | — | — | — | — | 42.2 | — | — | — | — | — | — | |
| PTGEN2019.10 | — | 36.44 | 15.66 | 33.42 | — | — | — | — | — | — | — | — | |
| PTGEN+COV2019.10 | — | 39.53 | 17.28 | 36.38 | — | — | — | — | — | — | — | — | |
| REFRESH2018.02 | — | 35 | 42 | 16 | 7 | 66.34 | — | — | — | — | — | — | |
| See et al. (2017)2018.02 | — | 14 | 18 | 31 | 36 | 28.73 | — | — | — | — | — | — | |
| SUMOType=Extractive2019.08 | — | — | — | — | — | 41.7 | — | — | — | — | — | — | |
| T5 BaseTraining Objective=Standard Cross-Entropy2023.09 | — | 40.7 | 18.4 | 27.4 | 37.6 | — | — | — | 29.3 | 34.9 | — | — | |
| T5 Base + SDSTraining Objective=SDS (Self-supervised Objective for Summarization)2023.09 | — | 41.6 | 19.2 | 28.3 | 38.5 | — | — | — | 30.3 | 35.2 | — | — | |
| TransformerType=Extractive2019.08 | — | — | — | — | — | 37.8 | — | — | — | — | — | — | |
| UniLM2019.10 | — | 43.33 | 20.21 | 40.51 | — | — | — | — | — | — | — | — |