Summarization on Xsum
27.1ROUGE-2ST-MoE
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| ST-MoE#params=268B2022.09 | 27.1 | — | — | — | — | — | — | — | |
| PEGASUS-2B (calibrated)#params=2B, beam size=10, training=SLiC2022.09 | 27.09 | 49.77 | 42.08 | — | — | — | — | — | |
| PEGASUS-2B (fine-tuned)#params=2B, beam size=52022.09 | 26.77 | 49.57 | 41.41 | — | — | — | — | — | |
| Selfmem2023.05 | 26.7 | 50.3 | 41.6 | — | — | — | — | — | |
| [87]Variant=PG2023.05 | 26 | 48.2 | 40.2 | — | — | — | — | — | |
| [55]2023.05 | 25.6 | 49.1 | 40.4 | — | — | — | — | — | |
| PGAre-ranker=BRIO2023.05 | 25.51 | 48.44 | 40.34 | — | 91.45 | — | — | — | |
| Beam Searchre-ranker=BRIO2023.05 | 25.5 | 48.4 | 40.36 | — | 91.46 | — | — | — | |
| BRIO-Ctrsource=reported in re-ranking papers2023.05 | 25.13 | 48.13 | 39.8 | — | — | — | — | — | |
| [68]2023.05 | 25 | 48.1 | 40 | — | — | — | — | — | |
| [16]2023.05 | 25 | 47.8 | 39.7 | — | — | — | — | — | |
| SummaReRankersource=reported in re-ranking papers2023.05 | 24.95 | 48.12 | 40 | — | — | — | — | — | |
| Diverse Beamre-ranker=BRIO2023.05 | 24.84 | 47.86 | 39.81 | — | 91.41 | — | — | — | |
| PEGA-CONTBackbone=PEGASUS (large)2022.05 | 24.69 | 47.76 | 39.46 | — | — | — | — | — | |
| M-RAGLLM=Mixtral 8 × 7B2024.05 | 24.66 | 48.13 | 39.43 | — | — | — | — | — | |
| Top Beamsetup=single candidate2023.05 | 24.6 | 47.23 | 39.37 | — | 91.32 | — | — | — | |
| [100]2023.05 | 24.6 | 47.2 | 39.3 | — | — | — | — | — | |
| [54]2023.05 | 24.6 | 47.6 | 39.4 | — | — | — | — | — | |
| Previous SOTA2022.05 | 24.57 | 47.61 | 39.44 | — | — | — | — | — | |
| SimCLSsource=reported in re-ranking papers2023.05 | 24.57 | 47.61 | 39.44 | — | — | — | — | — | |
| PEGASUSBackbone=large2022.05 | 24.56 | 47.21 | 39.25 | — | — | — | — | — | |
| [87]Variant=L2023.05 | 24.1 | 46.5 | 38.6 | — | — | — | — | — | |
| [69]2023.05 | 24.1 | 47.1 | 38.8 | — | — | — | — | — | |
| SummaFusionsource=reported in re-ranking papers2023.05 | 24.05 | 47.08 | 38.82 | — | — | — | — | — | |
| Nucleusre-ranker=BRIO2023.05 | 23.74 | 46.78 | 38.86 | — | 91.2 | — | — | — | |
| SelfmemLLM=Mixtral 8 × 7B2024.05 | 22.38 | 44.67 | 37.86 | — | — | — | — | — | |
| [40]2023.05 | 22.3 | 45.1 | 37.3 | — | — | — | — | — | |
| Self-RAGLLM=Mixtral 8 × 7B2024.05 | 22.26 | 44.01 | 37.51 | — | — | — | — | — | |
| NaiveLLM=Mixtral 8 × 7B2024.05 | 22.07 | 43.82 | 37.44 | — | — | — | — | — | |
| [87]Variant=B2023.05 | 21 | 43.1 | 35.5 | — | — | — | — | — | |
| BART#Bits (E-W-A)=32-32-32, Size (MB)=532.0, FLOPS=1x2023.06 | 20.79 | 43.84 | 35.71 | — | — | — | — | — | |
| HyperMoEArchitecture=Switch Transformer2024.02 | 19.67 | — | — | — | — | — | — | — | |
| DQ-BART#Bits (E-W-A)=8-8-8, Size (MB)=138.12023.06 | 19.61 | 42.51 | 34.61 | — | — | — | — | — | |
| TBT#Bits (E-W-A)=2-2-8, Size (MB)=39.6, FLOPS=0.25x2023.06 | 19.54 | 42.4 | 34.51 | — | — | — | — | — | |
| MoE-ShareArchitecture=Switch Transformer2024.02 | 19.41 | — | — | — | — | — | — | — | |
| MoEArchitecture=Switch Transformer2024.02 | 19.35 | — | — | — | — | — | — | — | |
| RoCoFT-ColumnBackbone=BART-large, Trainable Parameters=0.597M2024.10 | 18.54 | 40.62 | 32.17 | — | — | — | — | — | |
| RoCoFT-RowBackbone=BART-large, Trainable Parameters=0.597M2024.10 | 18.48 | 40.12 | 31.93 | — | — | — | — | — | |
| LoRABackbone=BART-large, Trainable Parameters=1.242M2024.10 | 18.44 | 40.63 | 32.15 | — | — | — | — | — | |
| TBT#Bits (E-W-A)=1-1-8, Size (MB)=23.2, FLOPS=0.125x2023.06 | 18.37 | 40.96 | 33.3 | — | — | — | — | — | |
| AdaLoRABackbone=BART-large, Trainable Parameters=1.663M2024.10 | 18.28 | 40.95 | 31.84 | — | — | — | — | — | |
| LoHaBackbone=BART-large, Trainable Parameters=1.285M2024.10 | 18.08 | 40.12 | 32.39 | — | — | — | — | — | |
| M-RAGLLM=Llama 2 13B2024.05 | 18.02 | 37.18 | 26.44 | — | — | — | — | — | |
| QuantBart#Bits (E-W-A)=8-8-8, Size (MB)=138.12023.06 | 17.78 | 40.25 | 32.7 | — | — | — | — | — | |
| DQ-BART#Bits (E-W-A)=2-2-8, Size (MB)=39.6, FLOPS=0.25x2023.06 | 17.34 | 40.06 | 32.46 | — | — | — | — | — | |
| Baseline (TWN)#Bits (E-W-A)=2-2-8, Size (MB)=39.6, FLOPS=0.25x2023.06 | 17.13 | 39.99 | 31.99 | — | — | — | — | — | |
| BinaryBert*#Bits (E-W-A)=1-1-8, Size (MB)=23.2, FLOPS=0.125x2023.06 | 17.05 | 39.76 | 31.99 | — | — | — | — | — | |
| T5-CONTBackbone=small2022.05 | 16.96 | 39.66 | 31.86 | — | — | — | — | — | |
| BitfitBackbone=BART-large, Trainable Parameters=0.672M2024.10 | 16.87 | 39.1 | 30.43 | — | — | — | — | — | |
| QuantBart#Bits (E-W-A)=2-2-8, Size (MB)=39.6, FLOPS=0.25x2023.06 | 16.72 | 39.15 | 31.72 | — | — | — | — | — | |
| PEGASUSBackbone=base2022.05 | 16.58 | 39.79 | 31.7 | — | — | — | — | — | |
| [51]2023.05 | 16.5 | 38.8 | 31.3 | — | — | — | — | — | |
| FTBackbone=BART-large, Trainable Parameters=460M2024.10 | 16.19 | 40.73 | 30.13 | — | — | — | — | — | |
| LoKrBackbone=BART-large, Trainable Parameters=1.089M2024.10 | 16.14 | 39.03 | 30.42 | — | — | — | — | — | |
| T5-CLAPSBackbone=small2022.05 | 15.78 | 37.89 | 30.59 | — | — | — | — | — | |
| T5-Naive CLBackbone=small, Loss=N-Pairs2022.05 | 15.48 | 37.76 | 30.15 | — | — | — | — | — | |
| Prefix-tuningBackbone=BART-large, Trainable Parameters=2.983M2024.10 | 15.16 | 38.24 | 28.84 | — | — | — | — | — | |
| T5-Dropout CLBackbone=small2022.05 | 14.93 | 36.82 | 29.26 | — | — | — | — | — | |
| T5-SSMBA CLBackbone=small2022.05 | 14.81 | 36.58 | 29.68 | — | — | — | — | — | |
| T5-Naive CLBackbone=small2022.05 | 14.81 | 36.34 | 29.41 | — | — | — | — | — | |
| T5Backbone=small2022.05 | 14.72 | 36.1 | 29.16 | — | — | — | — | — | |
| Prompt tuningBackbone=BART-large, Trainable Parameters=0.755M2024.10 | 14.46 | 38.24 | 27.89 | — | — | — | — | — | |
| TBT#Bits (E-W-A)=2-2-2, Size (MB)=39.6, FLOPS=0.0625x2023.06 | 14.38 | 36.21 | 29.07 | — | — | — | — | — | |
| PEGASUSFew-shot samples=1002022.11 | 13.69 | 37.3 | 29.08 | — | — | — | — | — | |
| BEST-ON-VALBackbone=Llama-22024.12 | 13 | — | — | — | — | — | — | — | |
| SMOOTHIE-GLOBALBackbone=Llama-22024.12 | 12.8 | — | — | — | — | — | — | — | |
| SMOOTHIE-LOCALBackbone=Llama-22024.12 | 12.7 | — | — | — | — | — | — | — | |
| PaLM 540BLearning Setting=One-Shot2022.05 | 12.2 | — | — | — | — | — | — | — | |
| M-RAGLLM=Phi-2 2.7B2024.05 | 11.57 | 30.7 | 26.2 | — | — | — | — | — | |
| UNISUMMFew-shot samples=1002022.11 | 11.36 | 33.33 | 25.85 | — | — | — | — | — | |
| PaLM 62BLearning Setting=One-Shot2022.05 | 11.2 | — | — | — | — | — | — | — | |
| RANDOMBackbone=Llama-22024.12 | 11.2 | — | — | — | — | — | — | — | |
| TBT#Bits (E-W-A)=1-1-1, Size (MB)=23.2, FLOPS=0.0156x2023.06 | 11.19 | 31.68 | 25.29 | — | — | — | — | — | |
| MultiBARTFew-shot samples=1002022.11 | 10.88 | 31.48 | 25 | — | — | — | — | — | |
| BART-PTFew-shot samples=1002022.11 | 9.87 | 29.7 | 23.7 | — | — | — | — | — | |
| GPT2-treeModel type=Baseline, Input=Text with silver parse trees2025.06 | 9.47 | 29.59 | 23.58 | 20.88 | — | — | — | — | |
| Nb-Dn-In-NmComposition=Non-binary, Direction=Downward, Scope=Interior, Masking=None2025.06 | 9.4 | 29.48 | 23.54 | 20.81 | — | — | — | — | |
| Nb-Up-In-NmComposition=Non-binary, Direction=Upward, Scope=Interior, Masking=None2025.06 | 9.06 | 29.05 | 23.21 | 20.44 | — | — | — | — | |
| Bi-Up-In-NmComposition=Binary, Direction=Upward, Scope=Interior, Masking=None2025.06 | 8.97 | 28.93 | 22.97 | 20.29 | — | — | — | — | |
| Bi-Up-Ex-NmComposition=Binary, Direction=Upward, Scope=Exterior, Masking=None2025.06 | 8.95 | 29.04 | 23.01 | 20.33 | — | — | — | — | |
| GPT-3.5 (text-davinci-002)Few-shot samples=12022.11 | 8.87 | — | — | — | — | — | — | — | |
| UL2 20BLearning Setting=One-Shot2022.05 | 8.6 | 25.5 | 19.8 | — | — | — | — | — | |
| PaLM 8BLearning Setting=One-Shot2022.05 | 7.9 | — | — | — | — | — | — | — | |
| GPT2-tokenModel type=Baseline2025.06 | 7.67 | 27.14 | 21.65 | 18.82 | — | — | — | — | |
| Nb-Dn-In-MComposition=Non-binary, Direction=Downward, Scope=Interior, Masking=Sub-constituent masking2025.06 | 7.31 | 26.1 | 20.98 | 18.07 | — | — | — | — | |
| BART-FTFew-shot samples=1002022.11 | 7.21 | 27.45 | 21.74 | — | — | — | — | — | |
| UNISUMMFew-shot samples=102022.11 | 7.2 | 26.1 | 19.92 | — | — | — | — | — | |
| BEST-ON-VALBackbone=Falcon2024.12 | 6.7 | — | — | — | — | — | — | — | |
| Bi-Up-In-MComposition=Binary, Direction=Upward, Scope=Interior, Masking=Sub-constituent masking2025.06 | 6.64 | 24.84 | 19.88 | 17.12 | — | — | — | — | |
| BART-FTFew-shot samples=102022.11 | 6.42 | 24.89 | 19.18 | — | — | — | — | — | |
| SMOOTHIE-GLOBALBackbone=Falcon2024.12 | 6.4 | — | — | — | — | — | — | — | |
| NoneLLM=Mixtral 8 × 7B2024.05 | 6.39 | 25.4 | 18.3 | — | — | — | — | — | |
| Nb-Up-In-MComposition=Non-binary, Direction=Upward, Scope=Interior, Masking=Sub-constituent masking2025.06 | 6.28 | 24.3 | 19.45 | 16.68 | — | — | — | — | |
| SMOOTHIE-LOCALBackbone=Falcon2024.12 | 6 | — | — | — | — | — | — | — | |
| MultiBARTFew-shot samples=102022.11 | 5.76 | 20.76 | 17.01 | — | — | — | — | — | |
| Bi-Up-Ex-MComposition=Binary, Direction=Upward, Scope=Exterior, Masking=Sub-constituent masking2025.06 | 5.75 | 23.48 | 18.84 | 16.02 | — | — | — | — | |
| LaMDA 137BLearning Setting=One-Shot2022.05 | 5.4 | — | — | — | — | — | — | — | |
| RANDOMBackbone=Falcon2024.12 | 4.7 | — | — | — | — | — | — | — | |
| PEGASUSFew-shot samples=102022.11 | 3.98 | 20.55 | 14.8 | — | — | — | — | — | |
| BART-PTFew-shot samples=102022.11 | 2.77 | 14.29 | 11.52 | — | — | — | — | — |