Summarization on bigPatent
91.85ROUGE-1OracleFrag
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| OracleFragType=Oracle2019.09 | 91.85 | 78.66 | 91.85 | |
| LongT5Model Size=xl, Input Length=16k, Attention=TGlobal2021.12 | 76.87 | 66.06 | 70.76 | |
| LongT5Model Size=xl, Input Length=8k, Attention=TGlobal2021.12 | 76.39 | 65.37 | 70.16 | |
| LongT5Model Size=xl, Input Length=4k, Attention=TGlobal2021.12 | 75.82 | 64.64 | 69.54 | |
| M-RAGLLM=Mixtral 8 × 7B2024.05 | 71.34 | 42.24 | 47.22 | |
| LongT5Model Size=large, Input Length=16k, Attention=TGlobal2021.12 | 70.38 | 56.81 | 62.73 | |
| LongT5Model Size=large, Input Length=8k, Attention=TGlobal2021.12 | 67.42 | 52.62 | 59.04 | |
| T5.1.1 + PSGModel Size=large, Input Length=4k2021.12 | 67.05 | 52.24 | 58.7 | |
| LongT5Model Size=large, Input Length=4k, Attention=TGlobal2021.12 | 66.17 | 51.1 | 57.7 | |
| T5.1.1 + PSGModel Size=large, Input Length=2k2021.12 | 64.51 | 49.15 | 56.01 | |
| SelfmemLLM=Mixtral 8 × 7B2024.05 | 64.12 | 39.21 | 46.21 | |
| Self-RAGLLM=Mixtral 8 × 7B2024.05 | 63.59 | 38.65 | 45.25 | |
| Selfmem2023.05 | 62.9 | 48.1 | 59.6 | |
| T5.1.1Model Size=large, Input Length=4k2021.12 | 62.14 | 45.85 | 52.95 | |
| LongT5Model Size=base, Input Length=4k, Attention=TGlobal2021.12 | 60.95 | 44.22 | 51.52 | |
| [87]Variant=L2023.05 | 60.7 | 43.3 | 51.3 | |
| BIGBIRD-PegasusModel Category=Large2020.07 | 60.64 | 42.46 | 50.01 | |
| BigBird-PEGASUSModel Size=large2021.12 | 60.64 | 42.46 | 50.01 | |
| [98]2023.05 | 60.6 | 42.5 | 50 | |
| M-RAGLLM=Llama 2 13B2024.05 | 60.31 | 37.33 | 33.47 | |
| NaiveLLM=Mixtral 8 × 7B2024.05 | 60.11 | 38.33 | 43.44 | |
| T5.1.1Model Size=large, Input Length=2k2021.12 | 60.07 | 43.49 | 50.9 | |
| [87]Variant=B2023.05 | 59.5 | 42.6 | 50.6 | |
| T5.1.1 + PSGModel Size=large, Input Length=1k2021.12 | 58.58 | 41.8 | 49.74 | |
| BIGBIRD-ROBERTaModel Category=Base2020.07 | 55.69 | 37.27 | 45.56 | |
| T5.1.1Model Size=large, Input Length=1k2021.12 | 55.07 | 37.49 | 45.9 | |
| [100]2023.05 | 53.6 | 33.1 | 42.3 | |
| [1]2023.05 | 52.3 | 33.5 | 42.8 | |
| PegasusModel Category=Large, Evaluation Source=Reported2020.07 | 52.29 | 33.08 | 41.75 | |
| PegasusModel Category=Large, Evaluation Source=Re-eval2020.07 | 52.25 | 33.04 | 41.8 | |
| NoneLLM=Mixtral 8 × 7B2024.05 | 47.41 | 16.63 | 25.14 | |
| [90]2023.05 | 45 | 20.3 | 39.2 | |
| [87]Variant=PG2023.05 | 44.6 | 21.5 | 33.3 | |
| [40]2023.05 | 44.4 | 21.3 | 31 | |
| Gold ExtType=Oracle2019.09 | 43.56 | 16.91 | 36.52 | |
| Transformer + PegasusModel Category=Base2020.07 | 43.55 | 20.43 | 31.8 | |
| Transformer + ROBERTAModel Category=Base2020.07 | 41.11 | 22.1 | 32.58 | |
| TLM+E (G,G)Type=Oracle2019.09 | 39.99 | 13.79 | 35.33 | |
| TransformerModel Category=Base2020.07 | 39.66 | 20.94 | 31.2 | |
| [65]2023.05 | 38.7 | 12.3 | 34.1 | |
| TLM+E (G,M)Type=Mix2019.09 | 38.65 | 12.31 | 34.09 | |
| Extr-Abst-TLMModel Category=Prior Art2020.07 | 38.65 | 12.31 | 34.09 | |
| Sent-rewritingType=Mix2019.09 | 37.12 | 11.87 | 32.45 | |
| TLMType=Abs2019.09 | 36.41 | 11.38 | 30.88 | |
| Sent-CLFType=Ext2019.09 | 36.2 | 10.99 | 31.83 | |
| Sent-CLFModel Category=Prior Art2020.07 | 36.2 | 10.99 | 31.83 | |
| TextRankType=Ext2019.09 | 35.99 | 11.14 | 29.6 | |
| LexRankType=Ext2019.09 | 35.57 | 10.47 | 29.03 | |
| LexRankModel Category=Prior Art2020.07 | 35.57 | 10.47 | 29.03 | |
| RNN-ExtType=Ext2019.09 | 34.63 | 10.62 | 29.43 | |
| Sent-PTRType=Ext2019.09 | 34.21 | 10.78 | 30.07 | |
| Sent-PTRModel Category=Prior Art2020.07 | 34.21 | 10.78 | 30.07 | |
| Pointer-gen (Cov)Type=Mix2019.09 | 33.14 | 11.63 | 28.55 | |
| Pntr-Gen-Seq2SeqModel Category=Prior Art2020.07 | 33.14 | 11.63 | 28.55 | |
| Lead-3Type=Ext2019.09 | 31.27 | 8.75 | 26.18 | |
| M-RAGLLM=Phi-2 2.7B2024.05 | 31.25 | 14.72 | 18.98 | |
| Pointer-genType=Mix2019.09 | 30.59 | 10.01 | 25.65 | |
| Seq2SeqType=Abs2019.09 | 28.74 | 7.87 | 24.66 | |
| Attn-Seq2SeqModel Category=Prior Art2020.07 | 28.74 | 7.87 | 24.66 | |
| SumBasicType=Ext2019.09 | 27.44 | 7.08 | 23.66 | |
| SumBasicModel Category=Prior Art2020.07 | 27.44 | 7.08 | 23.66 |