Summarization on SAMSum
91.3BERTScore F1CriSPO
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| CriSPOLLM=Claude 3 Sonnet, Prompt Strategy=Automatic, Shots=3-shot, Example Selection=true2024.10 | 91.3 | — | — | — | |
| CriSPOLLM=Claude Instant, Prompt Strategy=Automatic, Shots=3-shot, Example Selection=true2024.10 | 90.7 | — | — | — | |
| CriSPOLLM=Llama3 8B, Prompt Strategy=Automatic, Shots=3-shot, Example Selection=true, Optimizer=Claude 3 Sonnet2024.10 | 90.7 | — | — | — | |
| CriSPOLLM=Claude 3 Sonnet, Prompt Strategy=Automatic, Shots=0-shot2024.10 | 90.6 | — | — | — | |
| Self-Consistency (SC)Backbone=GPT-4.1, Shot-setting=2-shot2025.12 | 90.57 | — | — | 29.87 | |
| Iterative Refine (IR)Backbone=GPT-4.1, Shot-setting=2-shot2025.12 | 90.47 | — | — | 29.97 | |
| CriSPOLLM=Claude Instant, Prompt Strategy=Automatic, Shots=0-shot2024.10 | 90.4 | — | — | — | |
| VanillaBackbone=GPT-4.1, Shot-setting=2-shot2025.12 | 90.37 | — | — | 31.58 | |
| ManualLLM=Claude 3 Sonnet, Shots=3-shot, Example Selection=true2024.10 | 90.3 | — | — | — | |
| Cited Summarization (Cite)Backbone=GPT-4.1, Shot-setting=2-shot2025.12 | 90.11 | — | — | 27.49 | |
| CriSPOLLM=Mistral 7B, Prompt Strategy=Automatic, Shots=3-shot, Example Selection=true2024.10 | 90.1 | — | — | — | |
| ManualLLM=Mistral 7B, Shots=3-shot, Example Selection=true2024.10 | 90 | — | — | — | |
| OPROLLM=Llama3 8B, Prompt Strategy=Automatic, Optimizer=Claude 3 Sonnet2024.10 | 89.9 | — | — | — | |
| Question-Answer Guided (QAG)Backbone=GPT-4.1, Shot-setting=2-shot2025.12 | 89.89 | — | — | 27.73 | |
| ManualLLM=Claude Instant, Shots=3-shot, Example Selection=true2024.10 | 89.8 | — | — | — | |
| OPROLLM=Claude Instant, Prompt Strategy=Automatic2024.10 | 89.8 | — | — | — | |
| OPROLLM=Claude 3 Sonnet, Prompt Strategy=Automatic2024.10 | 89.8 | — | — | — | |
| OPROLLM=Mistral 7B, Prompt Strategy=Automatic2024.10 | 89.8 | — | — | — | |
| ManualLLM=Claude 3 Sonnet, Shots=0-shot2024.10 | 89.5 | — | — | — | |
| CriSPOLLM=Mistral 7B, Prompt Strategy=Automatic, Shots=0-shot2024.10 | 89.5 | — | — | — | |
| Chain-of-Thought (CoT)Backbone=GPT-4.1, Shot-setting=2-shot2025.12 | 89.41 | — | — | 29.62 | |
| Plan-then-Write (Plan)Backbone=GPT-4.1, Shot-setting=2-shot2025.12 | 89.34 | — | — | 23.71 | |
| Extract-to-Abstract (E2A)Backbone=GPT-4.1, Shot-setting=2-shot2025.12 | 89.33 | — | — | 28.44 | |
| ManualLLM=Claude Instant, Shots=0-shot2024.10 | 89.2 | — | — | — | |
| o1Backbone=Large Reasoning Model, Shot-setting=2-shot2025.12 | 89.02 | — | — | 25.5 | |
| ManualLLM=Llama3 8B, Shots=0-shot2024.10 | 88.7 | — | — | — | |
| ManualLLM=Mistral 7B, Shots=0-shot2024.10 | 88.3 | — | — | — | |
| GPT-5Backbone=Large Reasoning Model, Shot-setting=2-shot2025.12 | 87.67 | — | — | 25.58 | |
| o3Backbone=Large Reasoning Model, Shot-setting=2-shot2025.12 | 86.84 | — | — | 22.67 | |
| Decomposition (Deco)Backbone=GPT-4.1, Shot-setting=2-shot2025.12 | 86.02 | — | — | 25.27 | |
| BART-PTNumber of shots=102022.11 | — | 1.58 | — | — | |
| BART-PTNumber of shots=1002022.11 | — | 0.29 | — | — | |
| CriSPOLLM=Claude Instant, Few-shot setting=0-shot2024.10 | — | — | 44.4 | — | |
| CriSPOLLM=Claude Instant, Few-shot setting=3-shot2024.10 | — | — | 45.8 | — | |
| CriSPOLLM=Claude3 Sonnet, Few-shot setting=0-shot2024.10 | — | — | 43.4 | — | |
| CriSPOLLM=Claude3 Sonnet, Few-shot setting=3-shot2024.10 | — | — | 47.2 | — | |
| CriSPOLLM=Mistral 7B, Few-shot setting=0-shot2024.10 | — | — | 37.6 | — | |
| CriSPOLLM=Mistral 7B, Few-shot setting=3-shot2024.10 | — | — | 40 | — | |
| CriSPOLLM=Llama3 8B, Few-shot setting=0-shot, Optimizer=Claude-32024.10 | — | — | 44.8 | — | |
| ManualLLM=Claude Instant, Few-shot setting=0-shot2024.10 | — | — | 33.9 | — | |
| ManualLLM=Claude Instant, Few-shot setting=3-shot2024.10 | — | — | 37.8 | — | |
| ManualLLM=Claude3 Sonnet, Few-shot setting=0-shot2024.10 | — | — | 35.8 | — | |
| ManualLLM=Claude3 Sonnet, Few-shot setting=3-shot2024.10 | — | — | 41.1 | — | |
| ManualLLM=Mistral 7B, Few-shot setting=0-shot2024.10 | — | — | 32 | — | |
| ManualLLM=Mistral 7B, Few-shot setting=3-shot2024.10 | — | — | 39.5 | — | |
| ManualLLM=Llama3 8B, Few-shot setting=0-shot2024.10 | — | — | 35.7 | — | |
| MultiBARTNumber of shots=102022.11 | — | 1.91 | — | — | |
| MultiBARTNumber of shots=1002022.11 | — | 0.4 | — | — | |
| OPROLLM=Claude Instant, Few-shot setting=0-shot2024.10 | — | — | 38.1 | — | |
| OPROLLM=Claude3 Sonnet, Few-shot setting=0-shot2024.10 | — | — | 39 | — | |
| OPROLLM=Mistral 7B, Few-shot setting=0-shot2024.10 | — | — | 37.9 | — | |
| OPROLLM=Llama3 8B, Few-shot setting=0-shot, Optimizer=Claude-32024.10 | — | — | 39.3 | — | |
| PEGASUSNumber of shots=102022.11 | — | 1.61 | — | — | |
| PEGASUSNumber of shots=1002022.11 | — | 0.47 | — | — | |
| UNISUMMNumber of shots=102022.11 | — | 1.07 | — | — | |
| UNISUMMNumber of shots=1002022.11 | — | 0.47 | — | — |