Query-based Meeting Summarization on QMSum
14.734Overall ScoreSLM
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| SLMBackbone=Llama3-70B-Instruct, ratio=0.52025.02 | 14.734 | — | 3.7 | |
| H2OBackbone=Llama3-70B-Instruct, ratio=0.52025.02 | 14.36 | — | 3.712 | |
| Llama3 (Full)Backbone=Llama3-70B-Instruct2025.02 | 13.993 | — | — | |
| SnapKVBackbone=Llama3-70B-Instruct, ratio=0.52025.02 | 13.846 | — | 2.793 | |
| ReFreeKVBackbone=Llama3-70B-Instruct, k=12025.02 | 13.835 | — | 3.028 | |
| SnapKVBackbone=Llama3-8B-Instruct, ratio=0.52025.02 | 5.519 | — | 0.264 | |
| Llama3 (Full)Backbone=Llama3-8B-Instruct2025.02 | 5.458 | — | — | |
| SLMBackbone=Llama3-8B-Instruct, ratio=0.52025.02 | 5.454 | — | 0.234 | |
| H2OBackbone=Llama3-8B-Instruct, ratio=0.52025.02 | 5.398 | — | 0.249 | |
| ReFreeKVBackbone=Llama3-8B-Instruct, k=12025.02 | 5.33 | — | 0.241 | |
| GLATraining Protocol=Finetuned from Mistral 7B, Training Tokens=20B, Input Context Length=16K, Base Model=Mistral 7B2024.09 | — | 9 | — | |
| GSATraining Protocol=Finetuned from Mistral 7B, Training Tokens=20B, Input Context Length=16K, Base Model=Mistral 7B2024.09 | — | 10 | — | |
| MambaTraining Protocol=Trained from scratch, Training Tokens=>1T, Input Context Length=16K2024.09 | — | 0.8 | — | |
| MistralTraining Protocol=Trained from scratch, Training Tokens=>1T, Input Context Length=16K2024.09 | — | 5 | — | |
| RetNetTraining Protocol=Finetuned from Mistral 7B, Training Tokens=20B, Input Context Length=16K, Base Model=Mistral 7B2024.09 | — | 0 | — | |
| RWKV6Training Protocol=Trained from scratch, Training Tokens=>1T, Input Context Length=16K2024.09 | — | 1.1 | — |