Summarization on ACI-Bench
63.1ROUGE-1CriSPO
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CriSPOLLM=Claude 3 Sonnet, Few-shot Setting=3-shot2024.10 | 63.1 | 32.5 | 41 | — | — | |
| ManualLLM=Claude 3 Sonnet, Few-shot Setting=3-shot2024.10 | 59.1 | 30.1 | 38.6 | — | — | |
| CriSPOLLM=Claude Instant, Few-shot Setting=3-shot2024.10 | 58.2 | 26.7 | 35.3 | — | — | |
| CriSPOLLM=Llama 3 8B, Few-shot Setting=0-shot, Optimizer=Claude 3 Sonnet2024.10 | 56.2 | 22.8 | 29.9 | — | — | |
| OPROLLM=Llama 3 8B, Few-shot Setting=0-shot, Optimizer=Claude 3 Sonnet2024.10 | 54.2 | 22 | 29.3 | — | — | |
| CriSPOLLM=Claude 3 Sonnet, Few-shot Setting=0-shot2024.10 | 54 | 21.4 | 30.3 | — | — | |
| CriSPOLLM=Claude Instant, Few-shot Setting=0-shot2024.10 | 53 | 19.7 | 26.8 | — | — | |
| ManualLLM=Claude Instant, Few-shot Setting=3-shot2024.10 | 51.5 | 23.6 | 33.5 | — | — | |
| ManualLLM=Llama 3 8B, Few-shot Setting=0-shot2024.10 | 50.5 | 19.8 | 27.7 | — | — | |
| CriSPOLLM=Mistral 7B, Few-shot Setting=3-shot2024.10 | 50.3 | 18.7 | 26.2 | — | — | |
| CriSPOLLM=Mistral 7B, Few-shot Setting=0-shot2024.10 | 50.2 | 18.2 | 25.6 | — | — | |
| OPROLLM=Claude 3 Sonnet, Few-shot Setting=0-shot2024.10 | 48.8 | 20.1 | 29.5 | — | — | |
| ManualLLM=Mistral 7B, Few-shot Setting=3-shot2024.10 | 48.4 | 19.2 | 28.1 | — | — | |
| ManualLLM=Mistral 7B, Few-shot Setting=0-shot2024.10 | 47.8 | 17.7 | 25.4 | — | — | |
| ManualLLM=Claude 3 Sonnet, Few-shot Setting=0-shot2024.10 | 47.3 | 20.3 | 29.3 | — | — | |
| OPROLLM=Claude Instant, Few-shot Setting=0-shot2024.10 | 45.2 | 16.3 | 25.5 | — | — | |
| OPROLLM=Mistral 7B, Few-shot Setting=0-shot2024.10 | 45.1 | 17 | 25.2 | — | — | |
| ManualLLM=Claude Instant, Few-shot Setting=0-shot2024.10 | 43.8 | 16.9 | 26.1 | — | — | |
| CriSPOLLM=Claude Instant, Strategy=Automatic Prompt Engineering, Evaluation Protocol=0-shot2024.10 | — | — | — | 19.7 | — | |
| CriSPOLLM=Claude Instant, Strategy=Automatic Prompt Engineering, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 26.7 | — | |
| CriSPOLLM=Claude3 Sonnet, Strategy=Automatic Prompt Engineering, Evaluation Protocol=0-shot2024.10 | — | — | — | 21.4 | — | |
| CriSPOLLM=Claude3 Sonnet, Strategy=Automatic Prompt Engineering, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 32.5 | — | |
| CriSPOLLM=Mistral 7B, Strategy=Automatic Prompt Engineering, Evaluation Protocol=0-shot2024.10 | — | — | — | 18.2 | — | |
| CriSPOLLM=Mistral 7B, Strategy=Automatic Prompt Engineering, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 18.7 | — | |
| CriSPOLLM=Llama3 8B, Strategy=Automatic Prompt Engineering, Evaluation Protocol=0-shot, Optimizer=Claude3 Sonnet2024.10 | — | — | — | 22.8 | — | |
| CriSPOLLM=Claude Instant, Few-shot setting=0-shot2024.10 | — | — | — | — | 53 | |
| CriSPOLLM=Claude Instant, Few-shot setting=3-shot2024.10 | — | — | — | — | 58.2 | |
| CriSPOLLM=Claude3 Sonnet, Few-shot setting=0-shot2024.10 | — | — | — | — | 54 | |
| CriSPOLLM=Claude3 Sonnet, Few-shot setting=3-shot2024.10 | — | — | — | — | 63.1 | |
| CriSPOLLM=Mistral 7B, Few-shot setting=0-shot2024.10 | — | — | — | — | 50.2 | |
| CriSPOLLM=Mistral 7B, Few-shot setting=3-shot2024.10 | — | — | — | — | 50.3 | |
| CriSPOLLM=Llama3 8B, Few-shot setting=0-shot, Optimizer=Claude-32024.10 | — | — | — | — | 56.2 | |
| ManualLLM=Claude Instant, Evaluation Protocol=0-shot2024.10 | — | — | — | 16.9 | — | |
| ManualLLM=Claude Instant, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 23.6 | — | |
| ManualLLM=Claude3 Sonnet, Evaluation Protocol=0-shot2024.10 | — | — | — | 20.3 | — | |
| ManualLLM=Claude3 Sonnet, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 30.1 | — | |
| ManualLLM=Mistral 7B, Evaluation Protocol=0-shot2024.10 | — | — | — | 17.7 | — | |
| ManualLLM=Mistral 7B, Evaluation Protocol=3-shot (ICL with example selection)2024.10 | — | — | — | 19.2 | — | |
| ManualLLM=Llama3 8B, Evaluation Protocol=0-shot2024.10 | — | — | — | 19.8 | — | |
| ManualLLM=Claude Instant, Few-shot setting=0-shot2024.10 | — | — | — | — | 43.9 | |
| ManualLLM=Claude Instant, Few-shot setting=3-shot2024.10 | — | — | — | — | 51.5 | |
| ManualLLM=Claude3 Sonnet, Few-shot setting=0-shot2024.10 | — | — | — | — | 47.3 | |
| ManualLLM=Claude3 Sonnet, Few-shot setting=3-shot2024.10 | — | — | — | — | 59.1 | |
| ManualLLM=Mistral 7B, Few-shot setting=0-shot2024.10 | — | — | — | — | 47.8 | |
| ManualLLM=Mistral 7B, Few-shot setting=3-shot2024.10 | — | — | — | — | 48.4 | |
| ManualLLM=Llama3 8B, Few-shot setting=0-shot2024.10 | — | — | — | — | 50.5 | |
| OPROLLM=Claude Instant, Strategy=Automatic Prompt Engineering2024.10 | — | — | — | 16.3 | — | |
| OPROLLM=Claude3 Sonnet, Strategy=Automatic Prompt Engineering2024.10 | — | — | — | 20.1 | — | |
| OPROLLM=Mistral 7B, Strategy=Automatic Prompt Engineering2024.10 | — | — | — | 17 | — | |
| OPROLLM=Llama3 8B, Strategy=Automatic Prompt Engineering, Optimizer=Claude3 Sonnet2024.10 | — | — | — | 22 | — | |
| OPROLLM=Claude Instant, Few-shot setting=0-shot2024.10 | — | — | — | — | 45.2 | |
| OPROLLM=Claude3 Sonnet, Few-shot setting=0-shot2024.10 | — | — | — | — | 48.8 | |
| OPROLLM=Mistral 7B, Few-shot setting=0-shot2024.10 | — | — | — | — | 45.1 | |
| OPROLLM=Llama3 8B, Few-shot setting=0-shot, Optimizer=Claude-32024.10 | — | — | — | — | 54.2 |