Question Answering on Natural Questions
70.58EMCOMI
Evaluation Results
| Method | Links | |
|---|---|---|
| COMICompression Rate=2x, Backbone=Qwen2-7B-Instruct2026.02 | 70.58 | |
| COMICompression Rate=4x, Backbone=Qwen2-7B-Instruct2026.02 | 69.38 | |
| COMICompression Rate=8x, Backbone=Qwen2-7B-Instruct2026.02 | 64.22 | |
| COMICompression Rate=16x, Backbone=Qwen2-7B-Instruct2026.02 | 57.48 | |
| Activation BeaconCompression Rate=2x, Backbone=Qwen2-7B-Instruct2026.02 | 51.83 | |
| Activation BeaconCompression Rate=4x, Backbone=Qwen2-7B-Instruct2026.02 | 48.47 | |
| COMICompression Rate=32x, Backbone=Qwen2-7B-Instruct2026.02 | 47.53 | |
| Activation BeaconCompression Rate=8x, Backbone=Qwen2-7B-Instruct2026.02 | 41.43 | |
| PaLM 540BEvaluation Protocol=Few-shot, Number of Shots=642022.04 | 39.6 | |
| PaLM 2-Lprompting=1-shot2023.05 | 37.5 | |
| LLaMA2 SFT 7B# Shot=0-shot, Total Params=6.7B, Activated Params=6.7B, FLOPs per 4K Tokens=187.9T2024.01 | 35.2 | |
| DeepSeekMoE Chat 16B# Shot=0-shot, Total Params=16.4B, Activated Params=2.8B, FLOPs per 4K Tokens=74.4T2024.01 | 35.1 | |
| DeepSeek Chat 7B# Shot=0-shot, Total Params=6.9B, Activated Params=6.9B, FLOPs per 4K Tokens=183.5T2024.01 | 32.7 | |
| GLaM 62B/64EEvaluation Protocol=Few-shot, Number of Shots=12022.04 | 32.5 | |
| PaLM 2-Mprompting=1-shot2023.05 | 32 | |
| Camelidae-8x34B-proShots=0-shot2024.01 | 31.2 | |
| DeepSeekMoE 16B ChatShots=0-shot2024.01 | 30.7 | |
| Llama2 70B ChatShots=0-shot2024.01 | 30.6 | |
| PaLMprompting=1-shot2023.05 | 29.3 | |
| Qwen 72B ChatShots=0-shot2024.01 | 29.3 | |
| PaLM 540BEvaluation Protocol=1-shot2022.04 | 29.3 | |
| GLaM 62B/64EEvaluation Protocol=1-shot2022.04 | 26.3 | |
| LLaMA2 7B# Shot=5-shot, # Total Params=6.7B, # Activated Params=6.7B, FLOPs per 4K Tokens=187.9T, # Training Tokens=2T2024.01 | 25.5 | |
| DeepSeekMoE 16B# Shot=5-shot, # Total Params=16.4B, # Activated Params=2.8B, FLOPs per 4K Tokens=74.4T, # Training Tokens=2T2024.01 | 25.5 | |
| DeepSeekMoE 16B# Shot=5-shot, # Total Params=16.4B, # Activated Params=2.8B, FLOPs per 4K Tokens=74.4T, # Training Tokens=2T2024.01 | 25.5 | |
| PaLM 2-Sprompting=1-shot2023.05 | 25.3 | |
| DeepSeekMoE 145B# Shot=5-shot2024.01 | 25 | |
| GLaM 62B/64EEvaluation Protocol=0-shot2022.04 | 24.7 | |
| Yi 34B ChatShots=0-shot2024.01 | 23.7 | |
| DeepSeekMoE 142B (Half Activated)# Shot=5-shot2024.01 | 23.5 | |
| DeepSeek 67B (Dense)# Shot=5-shot2024.01 | 22.6 | |
| Mixtral 8x7B Inst.Shots=0-shot2024.01 | 22.5 | |
| LongLLMLinguaCompression Rate=4x, Backbone=Qwen2-7B-Instruct2026.02 | 22.49 | |
| DeepSeek 7B (Dense)# Shot=5-shot, # Total Params=6.9B, # Activated Params=6.9B, FLOPs per 4K Tokens=183.5T, # Training Tokens=2T2024.01 | 22.2 | |
| PaLM 540BEvaluation Protocol=0-shot2022.04 | 21.2 | |
| LongLLMLinguaCompression Rate=2x, Backbone=Qwen2-7B-Instruct2026.02 | 19.81 | |
| LongLLMLinguaCompression Rate=8x, Backbone=Qwen2-7B-Instruct2026.02 | 19.44 | |
| GShard 137B# Shot=5-shot2024.01 | 19 | |
| LLMLingua-2-largeCompression Rate=2x, Backbone=Qwen2-7B-Instruct2026.02 | 15.52 | |
| LongLLMLinguaCompression Rate=16x, Backbone=Qwen2-7B-Instruct2026.02 | 15.07 | |
| LLMLingua-2-largeCompression Rate=4x, Backbone=Qwen2-7B-Instruct2026.02 | 14.39 | |
| LongLLMLinguaCompression Rate=32x, Backbone=Qwen2-7B-Instruct2026.02 | 13.37 | |
| LLMLingua-2-largeCompression Rate=8x, Backbone=Qwen2-7B-Instruct2026.02 | 12.77 | |
| Activation BeaconCompression Rate=16x, Backbone=Qwen2-7B-Instruct2026.02 | 11.68 | |
| LLMLingua-2-largeCompression Rate=16x, Backbone=Qwen2-7B-Instruct2026.02 | 10.36 | |
| LLMLingua-2-largeCompression Rate=32x, Backbone=Qwen2-7B-Instruct2026.02 | 9.6 | |
| Activation BeaconCompression Rate=32x, Backbone=Qwen2-7B-Instruct2026.02 | 7.95 | |
| SnapKVCompression Rate=8x, Backbone=Qwen2-7B-Instruct2026.02 | 6.25 | |
| SnapKVCompression Rate=16x, Backbone=Qwen2-7B-Instruct2026.02 | 6.06 | |
| SnapKVCompression Rate=2x, Backbone=Qwen2-7B-Instruct2026.02 | 6.01 | |
| SnapKVCompression Rate=4x, Backbone=Qwen2-7B-Instruct2026.02 | 5.99 | |
| SnapKVCompression Rate=32x, Backbone=Qwen2-7B-Instruct2026.02 | 5.76 |