Abstractive Summarization on arXiv (test)
53.7R-1Claude Instant + SigExt
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Claude Instant + SigExtBackbone=Claude-Instant-12024.10 | 53.7 | — | — | — | 45.2 | 23.5 | |
| Claude Instant + GP-SigExtBackbone=Claude-Instant-12024.10 | 53.5 | — | — | — | 44.7 | 23.2 | |
| Claude InstantBackbone=Claude-Instant-12024.10 | 53.2 | — | — | — | 44.4 | 23.1 | |
| Claude Instant + 2-stageBackbone=Claude-Instant-12024.10 | 50.4 | — | — | — | 44 | 22.9 | |
| FACTORSUMBudget=Oracle, Content Guidance=Lead2022.05 | 49.69 | 19.32 | 44.85 | 166 | — | — | |
| FACTORSUMBudget=Fixed, Content Guidance=Model2022.05 | 49.32 | 20.27 | 44.76 | 165 | — | — | |
| FACTORSUMBudget=Oracle, Content Guidance=None2022.05 | 48.87 | 18.83 | 43.96 | 167 | — | — | |
| FACTORSUMBudget=Model, Content Guidance=Model2022.05 | 48.74 | 20.12 | 44.19 | 164 | — | — | |
| HEPOS2022.05 | 48.24 | 20.26 | 41.78 | — | — | — | |
| FACTORSUMBudget=Fixed, Content Guidance=Lead2022.05 | 48.05 | 19.05 | 43.49 | 165 | — | — | |
| FACTORSUMBudget=Fixed, Content Guidance=None2022.05 | 47.22 | 18.6 | 42.61 | 165 | — | — | |
| DYLE2022.05 | 46.41 | 17.95 | 41.54 | — | — | — | |
| FACTORSUMBudget=Model, Content Guidance=None2022.05 | 46.4 | 18.21 | 41.85 | 164 | — | — | |
| BigBird2022.05 | 46.15 | 18.6 | 41.46 | 164 | — | — | |
| Mistral-7B + SigExtBackbone=Mistral-7B-Instruct2024.10 | 45.2 | — | — | — | 43.6 | 24.2 | |
| DANCER2022.05 | 45.01 | 17.6 | 40.56 | — | — | — | |
| Mistral-7B + GP-SigExtBackbone=Mistral-7B-Instruct2024.10 | 44.7 | — | — | — | 42.8 | 23.8 | |
| PEGASUS2022.05 | 43.06 | 16.39 | 38.65 | 168 | — | — | |
| Mistral-7B + 2-stageBackbone=Mistral-7B-Instruct2024.10 | 41.9 | — | — | — | 39.5 | 22 | |
| Mistral-7BBackbone=Mistral-7B-Instruct2024.10 | 41.6 | — | — | — | 43.1 | 24.6 | |
| 0-shot SOTAMode=Zero-shot2024.10 | — | — | — | — | 34.6 | 18.3 |