Over-refusal evaluation on MMMU (test)
37Math ScorePrompt-based
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Prompt-basedModel=LLaVA-1.5-7b, Evaluation Protocol=LLM-as-Judgement2026.01 | 37 | 5 | 32.5 | |
| PersonaModel=LLaVA-1.5-7b, Evaluation Protocol=LLM-as-Judgement2026.01 | 35.5 | 27 | 45 | |
| Prompt-basedModel=LLaVA-1.5-7b, Evaluation Protocol=Human Evaluation2026.01 | 35 | 4.5 | 32 | |
| PersonaModel=LLaVA-1.5-7b, Evaluation Protocol=Human Evaluation2026.01 | 34.5 | 27.5 | 44 | |
| PersonaModel=LLaVA-1.5-13b, Evaluation Protocol=Human Evaluation2026.01 | 24.5 | 11.5 | 47.5 | |
| Fine-tuningModel=InstructBLIP, Evaluation Protocol=Human Evaluation2026.01 | 24 | 46.5 | 33 | |
| Fine-tuningModel=InstructBLIP, Evaluation Protocol=LLM-as-Judgement2026.01 | 24 | 46.5 | 32.5 | |
| PersonaModel=LLaVA-1.5-13b, Evaluation Protocol=LLM-as-Judgement2026.01 | 22 | 11 | 50 | |
| PersonaModel=InstructBLIP, Evaluation Protocol=Human Evaluation2026.01 | 21.5 | 22 | 30 | |
| PersonaModel=InstructBLIP, Evaluation Protocol=LLM-as-Judgement2026.01 | 21 | 21 | 27.5 | |
| PersonaModel=Idefics3, Evaluation Protocol=LLM-as-Judgement2026.01 | 13 | 2 | 6 | |
| CR-VLMModel=InstructBLIP, Evaluation Protocol=LLM-as-Judgement2026.01 | 12.5 | 3.5 | 9.5 | |
| CR-VLMModel=LLaVA-1.5-7b, Evaluation Protocol=LLM-as-Judgement2026.01 | 12 | 2 | 6.5 | |
| CR-VLMModel=Idefics3, Evaluation Protocol=LLM-as-Judgement2026.01 | 11.5 | 1.5 | 9.5 | |
| Prompt-basedModel=InstructBLIP, Evaluation Protocol=LLM-as-Judgement2026.01 | 11.5 | 9.5 | 8.5 | |
| CR-VLMModel=InstructBLIP, Evaluation Protocol=Human Evaluation2026.01 | 10.5 | 2.5 | 6 | |
| CR-VLMModel=LLaVA-1.5-7b, Evaluation Protocol=Human Evaluation2026.01 | 9.5 | 1.5 | 4 | |
| PersonaModel=Idefics3, Evaluation Protocol=Human Evaluation2026.01 | 9 | 7 | 8.5 | |
| Prompt-basedModel=InstructBLIP, Evaluation Protocol=Human Evaluation2026.01 | 9 | 6 | 5 | |
| CR-VLMModel=LLaVA-1.5-13b, Evaluation Protocol=LLM-as-Judgement2026.01 | 7.5 | 2.5 | 5 | |
| CR-VLMModel=LLaVA-1.5-13b, Evaluation Protocol=Human Evaluation2026.01 | 7 | 0 | 4 | |
| CR-VLMModel=Idefics3, Evaluation Protocol=Human Evaluation2026.01 | 6 | 0 | 5 | |
| Prompt-basedModel=Idefics3, Evaluation Protocol=LLM-as-Judgement2026.01 | 4.5 | 2.5 | 5.5 | |
| Fine-tuningModel=LLaVA-1.5-7b, Evaluation Protocol=LLM-as-Judgement2026.01 | 3.5 | 0 | 1.5 | |
| Prompt-basedModel=LLaVA-1.5-13b, Evaluation Protocol=LLM-as-Judgement2026.01 | 3 | 1.5 | 11.5 | |
| Fine-tuningModel=Idefics3, Evaluation Protocol=LLM-as-Judgement2026.01 | 3 | 0.5 | 7 | |
| Fine-tuningModel=LLaVA-1.5-7b, Evaluation Protocol=Human Evaluation2026.01 | 2 | 0 | 1 | |
| Prompt-basedModel=LLaVA-1.5-13b, Evaluation Protocol=Human Evaluation2026.01 | 1.5 | 1.5 | 10.5 | |
| Prompt-basedModel=Idefics3, Evaluation Protocol=Human Evaluation2026.01 | 1.5 | 2 | 3 | |
| Fine-tuningModel=LLaVA-1.5-13b, Evaluation Protocol=LLM-as-Judgement2026.01 | 1.5 | 0 | 2.5 | |
| Fine-tuningModel=LLaVA-1.5-13b, Evaluation Protocol=Human Evaluation2026.01 | 0.5 | 0 | 0 | |
| Fine-tuningModel=Idefics3, Evaluation Protocol=Human Evaluation2026.01 | 0 | 0.5 | 4 |