Summarization on MeetingBank in-domain (test)
48.64ROUGE-1LLMLingua-2
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| LLMLingua-2Tokens=970, Compression Ratio (1/τ)=3.1x, Target LLM=GPT-3.5-Turbo-0613, Backbone=xlm-roberta-large2024.03 | 48.64 | 17.37 | 22.96 | 34.24 | 88.27 | |
| LLMLingua-2-smallTokens=984, Compression Ratio (1/τ)=3.0x, Target LLM=GPT-3.5-Turbo-0613, Backbone=multilingual-BERT2024.03 | 48.33 | 17.41 | 23.07 | 34.36 | 88.77 | |
| OriginalTokens=3,003, Compression Ratio (1/τ)=1.0x, Target LLM=GPT-3.5-Turbo-06132024.03 | 47.28 | 22.34 | 26.66 | 35.15 | 88.96 | |
| Selective-ContextTokens=1,222, Compression Ratio (1/τ)=2.5x, Target LLM=GPT-3.5-Turbo-06132024.03 | 39.21 | 10.83 | 18.73 | 27.67 | 84.48 | |
| LLMLinguaTokens=1,176, Compression Ratio (1/τ)=2.5x, Target LLM=GPT-3.5-Turbo-06132024.03 | 37.98 | 8.94 | 14.08 | 26.58 | 86.42 | |
| LLMLingua-2Target LLM=Mistral-7B, Tokens=970, Speedup (1/τ)=3.0x2024.03 | 30.18 | — | — | — | — | |
| LLMLingua-2-smallTarget LLM=Mistral-7B, Tokens=984, Speedup (1/τ)=3.0x2024.03 | 29.93 | — | — | — | — | |
| Selective-ContextTarget LLM=Mistral-7B, Tokens=1,222, Speedup (1/τ)=2.5x2024.03 | 26.84 | — | — | — | — | |
| Original PromptTarget LLM=Mistral-7B, Tokens=3,0032024.03 | 26.26 | — | — | — | — | |
| LLMLinguaTarget LLM=Mistral-7B, Tokens=1,176, Speedup (1/τ)=2.5x2024.03 | 23.63 | — | — | — | — |