Multimodal Understanding on MMMU (Accuracy, Average performance retention)
59.63AccuracyRISE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| RISEBackbone=Qwen3-VL-8B-Instruct, Self-evolving steps=602026.05 | 59.63 | — | |
| RISEBackbone=Qwen3-VL-8B-Instruct, Self-evolving steps=402026.05 | 58.11 | — | |
| RISEBackbone=Qwen3-VL-8B-Instruct, Self-evolving steps=202026.05 | 55.54 | — | |
| RISEBackbone=Qwen2.5-VL-7B-Instruct, Self-evolving steps=602026.05 | 55.31 | — | |
| Base ModelBackbone=Qwen3-VL-8B-Instruct, Self-evolving steps=02026.05 | 55.31 | — | |
| RISEBackbone=Qwen2.5-VL-7B-Instruct, Self-evolving steps=202026.05 | 54.03 | — | |
| RISEBackbone=Qwen2.5-VL-7B-Instruct, Self-evolving steps=402026.05 | 54.03 | — | |
| GreedyBackbone=Qwen2.5-VL (7B)2026.04 | 52.56 | — | |
| Qwen2-VL 7BBackbone=Qwen2-VL 7B, Sparsity Level=0%, Pruning Method=Dense, Pruning Type=None2026.03 | 50.89 | 100 | |
| Base ModelBackbone=Qwen2.5-VL-7B-Instruct, Self-evolving steps=02026.05 | 49.71 | — | |
| ε-SamplingTemperature (τ)=0.7, Backbone=Qwen2.5-VL (7B)2026.04 | 49.67 | — | |
| η-SamplingTemperature (τ)=0.7, Backbone=Qwen2.5-VL (7B)2026.04 | 49.56 | — | |
| Temp’ OnlyTemperature (τ)=0.7, Backbone=Qwen2.5-VL (7B)2026.04 | 47.78 | — | |
| Top-kTemperature (τ)=0.7, Backbone=Qwen2.5-VL (7B)2026.04 | 47.78 | — | |
| Min-pTemperature (τ)=0.7, Backbone=Qwen2.5-VL (7B)2026.04 | 47.33 | — | |
| LLaVA-Qwen-7B (Instruct)Model=LLaVA-Qwen-7B (Instruct)2025.02 | 46.8 | — | |
| FullTraining Strategy=Full-Attention baseline2026.06 | 46.8 | — | |
| Top-p (nucleus)Temperature (τ)=0.7, Backbone=Qwen2.5-VL (7B)2026.04 | 46.56 | — | |
| Top-p (nucleus)Temperature (τ)=1.0, Backbone=Qwen2.5-VL (7B)2026.04 | 46.33 | — | |
| PRISM-Qwen-7B (Instruct)Model=PRISM-Qwen-7B (Instruct)2025.02 | 46.1 | — | |
| Top-kTemperature (τ)=1.0, Backbone=Qwen2.5-VL (7B)2026.04 | 46 | — | |
| LLaVA-Qwen-7B (Base)Model=LLaVA-Qwen-7B (Base)2025.02 | 46 | — | |
| MSA-PTTraining Strategy=from-scratch sparse pretraining2026.06 | 45.9 | — | |
| Temp’ OnlyTemperature (τ)=1.0, Backbone=Qwen2.5-VL (7B)2026.04 | 45.67 | — | |
| GreedyModel=Qwen2.5-VL (3B)2026.04 | 45.44 | — | |
| ε-SamplingTemperature (τ)=1.0, Backbone=Qwen2.5-VL (7B)2026.04 | 45.44 | — | |
| PRISM-Qwen-7B (Base)Model=PRISM-Qwen-7B (Base)2025.02 | 45.2 | — | |
| η-SamplingTemperature (τ)=1.0, Backbone=Qwen2.5-VL (7B)2026.04 | 44.67 | — | |
| MSA-CPTTraining Strategy=sparse continued pretraining2026.06 | 44.5 | — | |
| Min-pTemperature (τ)=2.0, Backbone=Qwen2.5-VL (7B)2026.04 | 44.33 | — | |
| Min-pTemperature (τ)=1.0, Backbone=Qwen2.5-VL (7B)2026.04 | 44.11 | — | |
| Temp' OnlyModel=Qwen2.5-VL (3B), tau=0.72026.04 | 43 | — | |
| η-SamplingModel=Qwen2.5-VL (3B), tau=0.72026.04 | 43 | — | |
| Top-p (nucleus)Model=Qwen2.5-VL (3B), tau=0.72026.04 | 42.69 | — | |
| Min-pModel=Qwen2.5-VL (3B), tau=0.72026.04 | 42.64 | — | |
| Top-kModel=Qwen2.5-VL (3B), tau=0.72026.04 | 42.53 | — | |
| ε-SamplingModel=Qwen2.5-VL (3B), tau=0.72026.04 | 42.45 | — | |
| ATV-PruningBackbone=Qwen2-VL 7B, Sparsity Level=60%, Pruning Method=ATV-Pruning, Pruning Type=unstructured, uniform pruning2026.03 | 41.56 | 85.65 | |
| Top-p (nucleus)Model=Qwen2.5-VL (3B), tau=1.02026.04 | 41.3 | — | |
| TinyLLaVA†LLM=Qwen2.5-3B2026.07 | 41.3 | — | |
| TAMPBackbone=Qwen2-VL 7B, Sparsity Level=60%, Pruning Method=TAMP, Pruning Type=unstructured, uniform pruning2026.03 | 41.22 | 83.79 | |
| Temp' OnlyModel=Qwen2.5-VL (3B), tau=1.02026.04 | 41.11 | — | |
| Top-kModel=Qwen2.5-VL (3B), tau=1.02026.04 | 41.08 | — | |
| SparseGPTBackbone=Qwen2-VL 7B, Sparsity Level=60%, Pruning Method=SparseGPT, Pruning Type=unstructured, uniform pruning2026.03 | 41 | 85.3 | |
| ε-SamplingModel=Qwen2.5-VL (3B), tau=1.02026.04 | 40.5 | — | |
| η-SamplingModel=Qwen2.5-VL (3B), tau=1.02026.04 | 40.44 | — | |
| Top-p (nucleus)Temperature (τ)=2.0, Backbone=Qwen2.5-VL (7B)2026.04 | 40.33 | — | |
| ε-SamplingTemperature (τ)=2.0, Backbone=Qwen2.5-VL (7B)2026.04 | 40.33 | — | |
| LLaVA-Llama-8BModel=LLaVA-Llama-8B2025.02 | 40.3 | — | |
| Temp’ OnlyTemperature (τ)=2.0, Backbone=Qwen2.5-VL (7B)2026.04 | 40.22 | — | |
| Min-pModel=Qwen2.5-VL (3B), tau=1.02026.04 | 40.2 | — | |
| LLaVA-NeXT 8BBackbone=LLaVA-NeXT 8B, Sparsity Level=0%, Pruning Method=Dense, Pruning Type=None2026.03 | 40.11 | 100 | |
| PRISM-Llama-8BModel=PRISM-Llama-8B2025.02 | 39.7 | — | |
| TinyLLaVA†LLM=Qwen2.5-1.5B2026.07 | 39.2 | — | |
| η-SamplingTemperature (τ)=2.0, Backbone=Qwen2.5-VL (7B)2026.04 | 39.11 | — | |
| TRAGLLM=Qwen2.5-1.5B2026.07 | 38.7 | — | |
| TinyLLaVA†LLM=Qwen1.5-4B2026.07 | 38.6 | — | |
| TinyLLaVALLM=Phi2-2.7B2026.07 | 38.4 | — | |
| Top-kTemperature (τ)=2.0, Backbone=Qwen2.5-VL (7B)2026.04 | 38.22 | — | |
| BunnyLLM=Phi2-2.7B2026.07 | 38.2 | — | |
| LLaVA-Phi2-3BModel=LLaVA-Phi2-3B2025.02 | 37.7 | — | |
| PRISM-3BModel=PRISM-3B2025.02 | 37.6 | — | |
| WandaBackbone=Qwen2-VL 7B, Sparsity Level=60%, Pruning Method=Wanda, Pruning Type=unstructured, uniform pruning2026.03 | 37.22 | 77.78 | |
| APETModel=LLaVA-NeXT, TFLOPs=70%2026.06 | 37.11 | 99.4 | |
| Ours (N=20)Model=LLaVA-NeXT, TFLOPs=67%2026.06 | 36.78 | 98.8 | |
| VSkip+ (N=20)Model=LLaVA-NeXT, TFLOPs=74%2026.06 | 36.22 | 98.2 | |
| ATV-PruningBackbone=LLaVA-NeXT 8B, Sparsity Level=50%, Pruning Method=ATV-Pruning, Pruning Type=unstructured, uniform pruning2026.03 | 36.11 | 94 | |
| SparseGPTBackbone=LLaVA-NeXT 8B, Sparsity Level=50%, Pruning Method=SparseGPT, Pruning Type=unstructured, uniform pruning2026.03 | 36 | 91.74 | |
| VanillaModel=LLaVA-NeXT, TFLOPs=100%2026.06 | 36 | 100 | |
| V2DropModel=LLaVA-NeXT, TFLOPs=70%2026.06 | 36 | 98.4 | |
| ShortV (N=20)Model=LLaVA-NeXT, TFLOPs=51%2026.06 | 35.89 | 96 | |
| LLaVA-KDLLM=Qwen2.5-1.5B2026.07 | 35.8 | — | |
| PRISM-13BModel=PRISM-13B2025.02 | 35.7 | — | |
| WandaBackbone=LLaVA-NeXT 8B, Sparsity Level=50%, Pruning Method=Wanda, Pruning Type=unstructured, uniform pruning2026.03 | 35.56 | 88.36 | |
| VTW (K=16)Model=LLaVA-NeXT, TFLOPs=51%2026.06 | 35.56 | 86.5 | |
| LLaVA-Vicuna-7BModel=LLaVA-Vicuna-7B2025.02 | 35.4 | — | |
| Full-Finetune2025.02 | 35.4 | — | |
| TRAGLLM=Qwen1.5-1.8B2026.07 | 35.2 | — | |
| LLaVA-Vicuna-13BModel=LLaVA-Vicuna-13B2025.02 | 35.1 | — | |
| ε-SamplingModel=Qwen2.5-VL (3B), tau=2.02026.04 | 35.03 | — | |
| TinyLLaVA†LLM=Qwen1.5-1.8B2026.07 | 34.8 | — | |
| TAMPBackbone=LLaVA-NeXT 8B, Sparsity Level=50%, Pruning Method=TAMP, Pruning Type=unstructured, uniform pruning2026.03 | 34.78 | 92.67 | |
| PRISM-7BModel=PRISM-7B2025.02 | 34.7 | — | |
| PRISM2025.02 | 34.7 | — | |
| CompoDistill†LLM=Qwen1.5-1.8B2026.07 | 34.1 | — | |
| CompoDistill†LLM=Qwen1.5-0.5B2026.07 | 34 | — | |
| Temp' OnlyModel=Qwen2.5-VL (3B), tau=2.02026.04 | 33.67 | — | |
| LLaVA-KDLLM=Qwen1.5-1.8B2026.07 | 33.6 | — | |
| Top-kModel=Qwen2.5-VL (3B), tau=2.02026.04 | 33.42 | — | |
| Length2025.02 | 33.1 | — | |
| KD†LLM=Qwen1.5-1.8B2026.07 | 32.7 | — | |
| KD†LLM=Qwen1.5-0.5B2026.07 | 32.3 | — | |
| TinyLLaVA†LLM=Qwen1.5-0.5B2026.07 | 31.9 | — | |
| Mini-Gemini-2BLLM=Gemma-2B2026.07 | 31.7 | — | |
| ATV-PruningBackbone=LLaVA-NeXT 8B, Sparsity Level=60%, Pruning Method=ATV-Pruning, Pruning Type=unstructured, uniform pruning2026.03 | 31.67 | 77.01 | |
| TRAGLLM=Qwen2.5-0.5B2026.07 | 31.3 | — | |
| TRAGLLM=Qwen1.5-0.5B2026.07 | 30.9 | — | |
| Random2025.02 | 30.5 | — | |
| TinyLLaVA†LLM=Qwen2.5-0.5B2026.07 | 30.4 | — | |
| LLaVA-KDLLM=Qwen1.5-0.5B2026.07 | 30.2 | — |