Multiple Choice QA and Reasoning on MMLU, SciQ, OQA, CQA, SIQA, PIQA, HellaSwag, WinoGrande, ARC-c, ARC-e
74.25MMLU AccuracyQwen2.5-7B
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7BBackbone=Qwen2.5-7B2025.09 | 74.25 | 97 | 52.8 | 84.52 | 58.44 | 81.72 | 80.24 | 77.27 | 63.82 | 87.21 | 75.73 | 100 | |
| SnapKVBackbone=Qwen2.5-7B2025.09 | 74.05 | 96.6 | 52.4 | 84.03 | 58.4 | 81.61 | 80.18 | 76.71 | 63.28 | 87.16 | 75.44 | 45.57 | |
| StreamingLLMBackbone=Qwen2.5-7B2025.09 | 73.31 | 96.6 | 52 | 84.28 | 58.19 | 81.39 | 79.76 | 76.64 | 62.79 | 86.36 | 75.13 | 45.14 | |
| SepLLMBackbone=Qwen2.5-7B2025.09 | 73.07 | 96.7 | 52.2 | 84.19 | 58.25 | 81.41 | 80.1 | 76.48 | 62.94 | 86.11 | 75.15 | 50.2 | |
| ChunkLLMBackbone=Qwen2.5-7B2025.09 | 72.51 | 96.6 | 52.4 | 84.68 | 58.34 | 81.77 | 80.08 | 76.87 | 63.65 | 87.21 | 75.41 | 45.47 | |
| Llama3.1-8BBackbone=Llama3.1-8B2025.09 | 65.3 | 97.6 | 48 | 74.28 | 54.04 | 83.19 | 81.76 | 80.03 | 57.85 | 84.55 | 72.66 | 100 | |
| ChunkLLMBackbone=Llama3.1-8B2025.09 | 64.78 | 97.3 | 48.4 | 74.45 | 54.76 | 82.75 | 81.84 | 79.01 | 57.68 | 84.55 | 72.55 | 45.04 | |
| SepLLMBackbone=Llama3.1-8B2025.09 | 64.32 | 97.4 | 47.4 | 74.1 | 54.25 | 83.03 | 81.68 | 79.01 | 57.93 | 84.09 | 72.32 | 50.32 | |
| SnapKVBackbone=Llama3.1-8B2025.09 | 62.57 | 97.7 | 47.2 | 74.37 | 54.09 | 82.75 | 81.66 | 79.72 | 57.34 | 84.8 | 72.49 | 45.72 | |
| StreamingLLMBackbone=Llama3.1-8B2025.09 | 61.19 | 97.2 | 48 | 73.79 | 53.94 | 81.56 | 80.14 | 78.22 | 56.91 | 83.59 | 72.45 | 45.3 |