Multimodal Question Answering on ScienceQA
97.8AccuracyCASHEW
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| CASHEWBackbone=InternVL3.5-8B2026.01 | 97.8 | — | — | |
| CASHEW-RLBackbone=Qwen3-VL-8B, Iterations (T)=32026.01 | 97.8 | — | — | |
| CASHEWBackbone=Qwen3-VL-8B2026.01 | 97.7 | — | — | |
| CASHEW-RLBackbone=Qwen3-VL-8B, Iterations (T)=12026.01 | 96.9 | — | — | |
| InternVL3.5-8BBackbone=InternVL3.5-8B2026.01 | 95.9 | — | — | |
| LLaVA-OneVision-7BBackbone=LLaVA-OneVision-7B2026.01 | 95.4 | — | — | |
| CASHEWBackbone=Qwen3-VL-4B2026.01 | 93.1 | — | — | |
| Qwen3-VL-8BBackbone=Qwen3-VL-8B2026.01 | 92.9 | — | — | |
| CASHEWBackbone=Qwen2.5-VL-7B2026.01 | 91.7 | — | — | |
| Qwen2.5-VL-7BBackbone=Qwen2.5-VL-7B2026.01 | 88.8 | — | — | |
| Upper boundBackbone=Janus-Pro-1B2025.12 | 86.28 | — | — | |
| Zero-shotBackbone=Janus-Pro-1B2025.12 | 82.69 | — | — | |
| MoDEBackbone=Janus-Pro-1B, Training Stage=Best Accuracy2025.12 | 81.01 | — | — | |
| VILA1.5-13BBackbone=VILA1.5-13B2026.01 | 79.1 | — | — | |
| CL-MoEBackbone=Janus-Pro-1B, Training Stage=Best Accuracy2025.12 | 74.17 | — | — | |
| LLaVA-Next-7BBackbone=LLaVA-Next-7B2026.01 | 73 | — | — | |
| TIVE2025.02 | 72.2 | — | — | |
| GraNd2025.02 | 71.4 | — | — | |
| PRISM2025.02 | 71.3 | — | — | |
| DataTailor2025.02 | 71 | — | — | |
| ICONS2025.02 | 70.8 | — | — | |
| EL2N2025.02 | 70.6 | — | — | |
| TIVE2025.02 | 70.6 | — | — | |
| Perplexity2025.02 | 70.5 | — | — | |
| VanillaBackbone=LLaVA-NeXT, Number of Visual Tokens=2880, Projector Fine-tuning=No2024.12 | 70.2 | 100 | — | |
| EL2N2025.02 | 70.2 | — | — | |
| Qwen3-VL-4BBackbone=Qwen3-VL-4B2026.01 | 69.5 | — | — | |
| Full-Finetune2025.02 | 69.4 | — | — | |
| COINCIDE2025.02 | 69.2 | — | — | |
| PRISM2025.02 | 69.1 | — | — | |
| GraNd2025.02 | 68.4 | — | — | |
| VisionZipBackbone=LLaVA-NeXT, Number of Visual Tokens=160, Projector Fine-tuning=No2024.12 | 68.3 | 97.3 | — | |
| Qwen-VL-Chat-7BBackbone=Qwen-VL-Chat-7B2026.01 | 68.2 | — | — | |
| VisionZipBackbone=LLaVA-NeXT, Number of Visual Tokens=640, Projector Fine-tuning=No2024.12 | 68.1 | 97 | — | |
| VisionZipBackbone=LLaVA-NeXT, Number of Visual Tokens=640, Projector Fine-tuning=Yes2024.12 | 67.9 | 96.7 | — | |
| Perplexity2025.02 | 67.9 | — | — | |
| SparseVLMBackbone=LLaVA-NeXT, Number of Visual Tokens=640, Projector Fine-tuning=No2024.12 | 67.7 | 96.4 | — | |
| VisionZipBackbone=LLaVA-NeXT, Number of Visual Tokens=320, Projector Fine-tuning=Yes2024.12 | 67.5 | 96.2 | — | |
| SparseVLMBackbone=LLaVA-NeXT, Number of Visual Tokens=160, Projector Fine-tuning=No2024.12 | 67.5 | 96.2 | — | |
| VisionZipBackbone=LLaVA-NeXT, Number of Visual Tokens=160, Projector Fine-tuning=Yes2024.12 | 67.5 | 96.2 | — | |
| SparseVLMBackbone=LLaVA-NeXT, Number of Visual Tokens=320, Projector Fine-tuning=No2024.12 | 67.3 | 95.9 | — | |
| VisionZipBackbone=LLaVA-NeXT, Number of Visual Tokens=320, Projector Fine-tuning=No2024.12 | 67.3 | 95.9 | — | |
| LLaVA-1.5-7BBackbone=LLaVA-1.5-7B2026.01 | 66.8 | — | — | |
| Length2025.02 | 66.8 | — | — | |
| Full-Finetune2025.02 | 66.8 | — | — | |
| Pretrained-gpt2-encoder-LLaVA1.5-7BLLM Backbone=Vicuna-7B, Vision Encoder Initialization=Pretrained (Language Bias), Model Scale=7B2026.04 | 66.7 | — | — | |
| Length2025.02 | 66.7 | — | — | |
| Scratch-gpt2-encoder-LLaVA1.5-7BLLM Backbone=Vicuna-7B, Vision Encoder Initialization=Scratch, Model Scale=7B2026.04 | 65.5 | — | — | |
| Random2025.02 | 65.5 | — | — | |
| MoDEBackbone=Janus-Pro-1B, Training Stage=Final Accuracy2025.12 | 65.45 | — | — | |
| Random2025.02 | 64.5 | — | — | |
| Model TailorBackbone=Janus-Pro-1B, Training Stage=Best Accuracy2025.12 | 61.58 | — | — | |
| Self-Filter2025.02 | 61.4 | — | — | |
| Seq LoRABackbone=Janus-Pro-1B, Training Stage=Best Accuracy2025.12 | 61.27 | — | — | |
| Seq LoRABackbone=Janus-Pro-1B, Training Stage=Final Accuracy2025.12 | 44.87 | — | — | |
| CL-MoEBackbone=Janus-Pro-1B, Training Stage=Final Accuracy2025.12 | 44.22 | — | — | |
| Model TailorBackbone=Janus-Pro-1B, Training Stage=Final Accuracy2025.12 | 38.62 | — | — | |
| Pretrained-gpt2-encoder-LLaVA1.6-13BLLM Backbone=Vicuna-13B, Vision Encoder Initialization=Pretrained (Language Bias), Model Scale=13B2026.04 | 24.6 | — | — | |
| Scratch-gpt2-encoder-LLaVA1.6-13BLLM Backbone=Vicuna-13B, Vision Encoder Initialization=Scratch, Model Scale=13B2026.04 | 22.5 | — | — | |
| Pretrained-gpt2-encoder-LLaVA1.6-7BLLM Backbone=Mistral-7B, Vision Encoder Initialization=Pretrained (Language Bias), Model Scale=7B2026.04 | 21 | — | — | |
| Scratch-gpt2-encoder-LLaVA1.6-7BLLM Backbone=Mistral-7B, Vision Encoder Initialization=Scratch, Model Scale=7B2026.04 | 17.7 | — | — | |
| 8-bit COAPTraining Time=7.8 h, Optimizer Mem. (GB)=4.9, Model Mem. (GB)=13.22024.11 | — | — | 92 | |
| 8-bit GaLoreTraining Time=30.0 h, Optimizer Mem. (GB)=4.9, Model Mem. (GB)=13.22024.11 | — | — | 90.7 | |
| COAPTraining Time=7.6 h, Optimizer Mem. (GB)=8.1, Model Mem. (GB)=13.22024.11 | — | — | 92.3 | |
| DeepSpeedTraining Time=47.1 h, Optimizer Mem. (GB)=26.4, Model Mem. (GB)=13.22024.11 | — | — | 82.4 | |
| FloraTraining Time=9.5 h, Optimizer Mem. (GB)=8.1, Model Mem. (GB)=13.22024.11 | — | — | 66.6 | |
| GaLoreTraining Time=30.2 h, Optimizer Mem. (GB)=8.1, Model Mem. (GB)=13.22024.11 | — | — | 91.1 | |
| LoRATraining Time=11.1 h, Optimizer Mem. (GB)=8.1, Model Mem. (GB)=17.22024.11 | — | — | 92.3 |