Document Summarization on GovReport (test)
75.56ROUGE-1ORACLE
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ORACLE2021.07 | 75.56 | 45.91 | 72.51 | — | — | |
| BART-LS# Param=440M, Pipelined approach=false2022.09 | 62 | 30.9 | 59.2 | — | — | |
| DYLE2022.05 | 61.01 | 28.83 | 57.82 | — | — | |
| DYLE# Param=525M, Pipelined approach=true2022.09 | 61 | 28.8 | 57.8 | — | — | |
| FACTORSUMBudget=Oracle, Content Guidance=Lead2022.05 | 60.73 | 25.24 | 57.2 | — | 650 | |
| SimCASBase Model=BARTlarge2023.08 | 60.29 | 26.68 | 27.97 | 68.64 | — | |
| FACTORSUMBudget=Fixed, Content Guidance=Model2022.05 | 60.1 | 25.28 | 56.65 | — | 648 | |
| PageSum# Param=440M, Pipelined approach=true2022.09 | 59.9 | 27.2 | 57.1 | — | — | |
| FACTORSUMBudget=Oracle, Content Guidance=None2022.05 | 59.8 | 24.13 | 56.12 | — | 651 | |
| FACTORSUMBudget=Fixed, Content Guidance=Lead2022.05 | 59.67 | 25.02 | 56.22 | — | 649 | |
| MemSumtype=extractive2021.07 | 59.43 | 28.6 | 56.69 | — | — | |
| LED# Param=460M, Pipelined approach=true2022.09 | 59.4 | 26.5 | 56.6 | — | — | |
| SimCASBase Model=BARTbase2023.08 | 59.3 | 25.95 | 27.07 | 68.17 | — | |
| NeuSumtype=extractive2021.07 | 58.94 | 25.38 | 55.8 | — | — | |
| FACTORSUMBudget=Model, Content Guidance=Model2022.05 | 58.78 | 24.87 | 55.37 | — | 638 | |
| FACTORSUMBudget=Fixed, Content Guidance=None2022.05 | 58.77 | 23.99 | 55.19 | — | 650 | |
| SLEDBackbone=BART-large, Parameters=406M2025.01 | 57.5 | 26.3 | 27.4 | 66.9 | — | |
| SLEDBase Model=BARTlarge2023.08 | 57.5 | 26.3 | 27.4 | — | — | |
| UnlimiformerBackbone=PRIMERA, Parameters=447M2025.01 | 57.4 | 26.2 | 28 | 68.1 | — | |
| UnlimiformerBase model=PRIMERA, Training strategy=alternating training2023.05 | 57.4 | 26.2 | 28 | 68.1 | — | |
| UnlimiformerBase Model=PRIMERA2023.08 | 57.4 | 26.2 | 28 | 68.1 | — | |
| FACTORSUMBudget=Model, Content Guidance=None2022.05 | 57.18 | 23.34 | 53.66 | — | 638 | |
| CachEDBackbone=BART-large, Parameters=406M2025.01 | 57 | 26.3 | 28.19 | 67.3 | — | |
| Memorizing transformersBase model=PRIMERA2023.05 | 57 | 25.3 | 26.5 | 67.7 | — | |
| BART-Hepos# Param=440M, Pipelined approach=true2022.09 | 56.9 | 22.6 | 53.8 | — | — | |
| Hepos-Sinkhorntype=abstractive2021.07 | 56.86 | 22.62 | 53.82 | — | — | |
| HEPOS2022.05 | 56.86 | 22.62 | 53.82 | — | — | |
| CachEDBackbone=BART-base, Parameters=139M2025.01 | 56.8 | 26.3 | 27.8 | 67 | — | |
| SUMMN2022.05 | 56.77 | 23.25 | 53.9 | — | — | |
| UnlimiformerBackbone=BART-base, Parameters=139M2025.01 | 56.6 | 26.3 | 27.6 | 68.2 | — | |
| UnlimiformerBase model=BART-base, Training strategy=alternating training2023.05 | 56.6 | 26.3 | 27.6 | 68.2 | — | |
| UnlimiformerBase Model=BARTbase2023.08 | 56.6 | 26.3 | 27.6 | 68.2 | — | |
| SFTBackbone=LED-base, Parameters=162M2025.01 | 56.3 | 25.8 | 27.4 | 65.6 | — | |
| Memorizing transformersBase model=BART-base2023.05 | 55.2 | 25.1 | 26.4 | 67.5 | — | |
| MemorizingBase Model=BARTbase2023.08 | 55.2 | 25.1 | 26.4 | 67.5 | — | |
| Standard finetuningBase model=PRIMERA2023.05 | 55.1 | 23.9 | 25.9 | 67 | — | |
| PRIMERABase Model=LEDlarge2023.08 | 55.1 | 23.9 | 25.9 | 67 | — | |
| Hepos-LSHtype=abstractive2021.07 | 55 | 21.13 | 51.67 | — | — | |
| SLEDBackbone=BART-base, Parameters=139M2025.01 | 54.7 | 24.4 | 25.4 | 67 | — | |
| SLEDBase model=BART-base2023.05 | 54.7 | 24.4 | 25.4 | 67 | — | |
| SLEDBase Model=BARTbase2023.08 | 54.7 | 24.4 | 25.4 | — | — | |
| SFTBackbone=BART-large, Parameters=406M2025.01 | 54.4 | 21.1 | 25.1 | 65.6 | — | |
| StandardBase Model=BARTlarge2023.08 | 52.94 | 19.78 | 23.71 | 64.44 | — | |
| BART-large2022.05 | 52.82 | 19.12 | 49.99 | — | 596 | |
| CachEDBackbone=T5-large, Parameters=770M2025.01 | 51.9 | 22.5 | 24.5 | 63.4 | — | |
| StandardBase Model=BARTbase2023.08 | 51.72 | 19.37 | 23.11 | 64.12 | — | |
| Leadtype=extractive2021.07 | 50.94 | 19.53 | 48.45 | — | — | |
| SFTBackbone=BART-base, Parameters=139M2025.01 | 50.2 | 19 | 23.7 | 63.4 | — | |
| Standard finetuningBase model=BART-base2023.05 | 48.7 | 19.2 | 22.8 | 64.3 | — | |
| SFTBackbone=T5-large, Parameters=770M2025.01 | 45.3 | 18.8 | 22.4 | 61.7 | — |