Summarization on MeetingBank
54.1ROUGE-LCriSPO
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CriSPOLLM=Claude 3 Sonnet, Few-shot Setting=3-shot2024.10 | 54.1 | 58.5 | 46.5 | — | — | |
| CriSPOLLM=Claude Instant, Few-shot Setting=3-shot2024.10 | 44.4 | 50.1 | 35.4 | — | — | |
| CriSPOLLM=Claude 3 Sonnet, Few-shot Setting=0-shot2024.10 | 40.9 | 47.4 | 32.5 | — | — | |
| CriSPOLLM=Llama 3 8B, Few-shot Setting=0-shot, Optimizer=Claude 3 Sonnet2024.10 | 36.8 | 44.7 | 27.6 | — | — | |
| ReplayRound Learned=R2, Task Type=Robust (NLP)2026.01 | 33.83 | — | — | — | — | |
| CriSPOLLM=Claude Instant, Few-shot Setting=0-shot2024.10 | 33.1 | 41.4 | 23.7 | — | — | |
| OPROLLM=Claude 3 Sonnet, Few-shot Setting=0-shot2024.10 | 32 | 41.5 | 21.8 | — | — | |
| OPROLLM=Llama 3 8B, Few-shot Setting=0-shot, Optimizer=Claude 3 Sonnet2024.10 | 31.5 | 40.2 | 22.3 | — | — | |
| OPROLLM=Claude Instant, Few-shot Setting=0-shot2024.10 | 29.7 | 39 | 20.3 | — | — | |
| SeqRound Learned=R2, Task Type=Robust (NLP)2026.01 | 29.68 | — | — | — | — | |
| ManualLLM=Claude 3 Sonnet, Few-shot Setting=3-shot2024.10 | 29.5 | 37.5 | 22 | — | — | |
| CriSPOLLM=Mistral 7B, Few-shot Setting=0-shot2024.10 | 29.3 | 39.1 | 19.5 | — | — | |
| AblationRound Learned=R2, Task Type=Robust (NLP)2026.01 | 29.04 | — | — | — | — | |
| OSWRound Learned=R2, Task Type=Robust (NLP)2026.01 | 26.99 | — | — | — | — | |
| CriSPOLLM=Mistral 7B, Few-shot Setting=3-shot2024.10 | 26.1 | 35.2 | 16.7 | — | — | |
| ManualLLM=Claude Instant, Few-shot Setting=3-shot2024.10 | 25.5 | 34.2 | 17.3 | — | — | |
| OPROLLM=Mistral 7B, Few-shot Setting=0-shot2024.10 | 24.2 | 33.9 | 15.4 | — | — | |
| ManualLLM=Mistral 7B, Few-shot Setting=3-shot2024.10 | 22.7 | 31.3 | 14.8 | — | — | |
| ManualLLM=Llama 3 8B, Few-shot Setting=0-shot2024.10 | 22.6 | 31.4 | 14.6 | — | — | |
| ManualLLM=Claude 3 Sonnet, Few-shot Setting=0-shot2024.10 | 22.3 | 31.2 | 14.2 | — | — | |
| ManualLLM=Claude Instant, Few-shot Setting=0-shot2024.10 | 20.5 | 30.7 | 11.6 | — | — | |
| ManualLLM=Mistral 7B, Few-shot Setting=0-shot2024.10 | 18.5 | 26 | 11.5 | — | — | |
| CriSPOLLM=Claude Instant, Strategy=Automatic Prompt Engineering, Evaluation Protocol=0-shot2024.10 | — | — | — | 23.7 | — | |
| CriSPOLLM=Claude Instant, Strategy=Automatic Prompt Engineering, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 35.4 | — | |
| CriSPOLLM=Claude3 Sonnet, Strategy=Automatic Prompt Engineering, Evaluation Protocol=0-shot2024.10 | — | — | — | 32.5 | — | |
| CriSPOLLM=Claude3 Sonnet, Strategy=Automatic Prompt Engineering, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 46.5 | — | |
| CriSPOLLM=Mistral 7B, Strategy=Automatic Prompt Engineering, Evaluation Protocol=0-shot2024.10 | — | — | — | 19.5 | — | |
| CriSPOLLM=Mistral 7B, Strategy=Automatic Prompt Engineering, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 16.7 | — | |
| CriSPOLLM=Llama3 8B, Strategy=Automatic Prompt Engineering, Evaluation Protocol=0-shot, Optimizer=Claude3 Sonnet2024.10 | — | — | — | 27.6 | — | |
| CriSPOLLM=Claude Instant, Few-shot setting=0-shot2024.10 | — | — | — | — | 41.4 | |
| CriSPOLLM=Claude Instant, Few-shot setting=3-shot2024.10 | — | — | — | — | 50.1 | |
| CriSPOLLM=Claude3 Sonnet, Few-shot setting=0-shot2024.10 | — | — | — | — | 47.4 | |
| CriSPOLLM=Claude3 Sonnet, Few-shot setting=3-shot2024.10 | — | — | — | — | 58.5 | |
| CriSPOLLM=Mistral 7B, Few-shot setting=0-shot2024.10 | — | — | — | — | 39.1 | |
| CriSPOLLM=Mistral 7B, Few-shot setting=3-shot2024.10 | — | — | — | — | 35.2 | |
| CriSPOLLM=Llama3 8B, Few-shot setting=0-shot, Optimizer=Claude-32024.10 | — | — | — | — | 44.7 | |
| ManualLLM=Claude Instant, Evaluation Protocol=0-shot2024.10 | — | — | — | 11.6 | — | |
| ManualLLM=Claude Instant, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 17.3 | — | |
| ManualLLM=Claude3 Sonnet, Evaluation Protocol=0-shot2024.10 | — | — | — | 14.2 | — | |
| ManualLLM=Claude3 Sonnet, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 22 | — | |
| ManualLLM=Mistral 7B, Evaluation Protocol=0-shot2024.10 | — | — | — | 11.5 | — | |
| ManualLLM=Mistral 7B, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 14.8 | — | |
| ManualLLM=Llama3 8B, Evaluation Protocol=0-shot2024.10 | — | — | — | 14.6 | — | |
| ManualLLM=Claude Instant, Few-shot setting=0-shot2024.10 | — | — | — | — | 30.7 | |
| ManualLLM=Claude Instant, Few-shot setting=3-shot2024.10 | — | — | — | — | 34.2 | |
| ManualLLM=Claude3 Sonnet, Few-shot setting=0-shot2024.10 | — | — | — | — | 31.2 | |
| ManualLLM=Claude3 Sonnet, Few-shot setting=3-shot2024.10 | — | — | — | — | 37.5 | |
| ManualLLM=Mistral 7B, Few-shot setting=0-shot2024.10 | — | — | — | — | 26 | |
| ManualLLM=Mistral 7B, Few-shot setting=3-shot2024.10 | — | — | — | — | 31.3 | |
| ManualLLM=Llama3 8B, Few-shot setting=0-shot2024.10 | — | — | — | — | 31.4 | |
| OPROLLM=Claude Instant, Strategy=Automatic Prompt Engineering2024.10 | — | — | — | 20.3 | — | |
| OPROLLM=Claude3 Sonnet, Strategy=Automatic Prompt Engineering2024.10 | — | — | — | 21.8 | — | |
| OPROLLM=Mistral 7B, Strategy=Automatic Prompt Engineering2024.10 | — | — | — | 15.4 | — | |
| OPROLLM=Llama3 8B, Strategy=Automatic Prompt Engineering, Optimizer=Claude3 Sonnet2024.10 | — | — | — | 22.3 | — | |
| OPROLLM=Claude Instant, Few-shot setting=0-shot2024.10 | — | — | — | — | 39 | |
| OPROLLM=Claude3 Sonnet, Few-shot setting=0-shot2024.10 | — | — | — | — | 41.5 | |
| OPROLLM=Mistral 7B, Few-shot setting=0-shot2024.10 | — | — | — | — | 33.9 | |
| OPROLLM=Llama3 8B, Few-shot setting=0-shot, Optimizer=Claude-32024.10 | — | — | — | — | 40.2 |