Multimodal Question Answering on ScienceQA v1.3 (test)
0.9019NAT ScoreFull Precision (Baseline)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Full Precision (Baseline)Backbone=LLaVA-13B, Precision=FP2024.10 | 0.9019 | 0.9314 | 0.8709 | 0.8939 | 0.8706 | 0.8983 | 0.9 | |
| Q-VLMBackbone=LLaVA-13B, Bits=W6A62024.10 | 0.8954 | 0.9318 | 0.865 | 0.8812 | 0.8701 | 0.8885 | 0.897 | |
| Q-VLMBackbone=LLaVA-7B, Bits=W6A62024.10 | 0.8943 | 0.9573 | 0.84 | 0.8871 | 0.8751 | 0.8725 | 0.8934 | |
| Full Precision (Baseline)Backbone=LLaVA-7B, Precision=FP2024.10 | 0.8939 | 0.9606 | 0.8564 | 0.8871 | 0.8765 | 0.885 | 0.8981 | |
| QLoRABackbone=LLaVA-13B, Bits=W6A62024.10 | 0.8887 | 0.9289 | 0.8564 | 0.8759 | 0.8656 | 0.8753 | 0.8887 | |
| QLoRABackbone=LLaVA-7B, Bits=W6A62024.10 | 0.8845 | 0.9471 | 0.8445 | 0.8763 | 0.8607 | 0.8787 | 0.8843 | |
| AWQBackbone=LLaVA-13B, Bits=W6A62024.10 | 0.8803 | 0.926 | 0.84 | 0.8602 | 0.8518 | 0.8641 | 0.8757 | |
| AWQBackbone=LLaVA-7B, Bits=W6A62024.10 | 0.8539 | 0.9201 | 0.8327 | 0.848 | 0.8354 | 0.8599 | 0.8623 | |
| Q-VLMBackbone=LLaVA-13B, Bits=W4A42024.10 | 0.8255 | 0.7332 | 0.8318 | 0.8103 | 0.7082 | 0.8674 | 0.8078 | |
| Q-VLMBackbone=LLaVA-7B, Bits=W4A42024.10 | 0.8086 | 0.7593 | 0.8073 | 0.8001 | 0.7248 | 0.839 | 0.7979 | |
| AWQBackbone=LLaVA-13B, Bits=W4A42024.10 | 0.8071 | 0.7061 | 0.7849 | 0.7946 | 0.7076 | 0.8182 | 0.7791 | |
| QLoRABackbone=LLaVA-13B, Bits=W4A42024.10 | 0.7962 | 0.7143 | 0.8245 | 0.7825 | 0.6842 | 0.853 | 0.7864 | |
| QLoRABackbone=LLaVA-7B, Bits=W4A42024.10 | 0.7753 | 0.7548 | 0.7918 | 0.7664 | 0.707 | 0.8195 | 0.7753 | |
| AWQBackbone=LLaVA-7B, Bits=W4A42024.10 | 0.7433 | 0.7222 | 0.7482 | 0.7341 | 0.6713 | 0.7798 | 0.7402 | |
| Q-VLMBackbone=MoE-LLaVA-1.6B, Bits=W6A62024.10 | 0.6414 | 0.5868 | 0.6285 | 0.628 | 0.5523 | 0.6669 | 0.6246 | |
| Full Precision (Baseline)Backbone=MoE-LLaVA-1.6B, Precision=FP2024.10 | 0.6401 | 0.5857 | 0.633 | 0.628 | 0.5478 | 0.6697 | 0.6268 | |
| QLoRABackbone=MoE-LLaVA-1.6B, Bits=W6A62024.10 | 0.6298 | 0.5778 | 0.6185 | 0.6187 | 0.5399 | 0.6537 | 0.616 | |
| AWQBackbone=MoE-LLaVA-1.6B, Bits=W6A62024.10 | 0.5983 | 0.5778 | 0.6148 | 0.5894 | 0.5295 | 0.6425 | 0.5983 | |
| Q-VLMBackbone=MoE-LLaVA-1.6B, Bits=W4A42024.10 | 0.5506 | 0.5194 | 0.5627 | 0.5457 | 0.4803 | 0.5834 | 0.5472 | |
| QLoRABackbone=MoE-LLaVA-1.6B, Bits=W4A42024.10 | 0.5448 | 0.4969 | 0.5555 | 0.532 | 0.473 | 0.5758 | 0.5324 | |
| AWQBackbone=MoE-LLaVA-1.6B, Bits=W4A42024.10 | 0.5369 | 0.4958 | 0.5427 | 0.5252 | 0.4781 | 0.5765 | 0.5298 |