Multimodal In-Context Learning on VL-ICL benchmark
77.6Average AccuracyMM1.5-30B
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| MM1.5-30BModel Size Class=Larger (>30B), Shot Count=4-shot2024.09 | 77.6 | 46.5 | 66.5 | 100 | 65 | 80 | 44.5 | |
| GPT-4VModel Size Class=Larger (>30B), Shot Count=4-shot2024.09 | 65.8 | 42 | 81 | 56 | 92 | 74 | 50 | |
| MM1.5-3B-MoEModel Size Class=3B, Shot Count=4-shot2024.09 | 59.6 | 32 | 58 | 92 | 63.3 | 65 | 47.5 | |
| MM1.5-3BModel Size Class=3B, Shot Count=4-shot2024.09 | 56.3 | 33.5 | 59 | 88 | 48.3 | 66.7 | 42.5 | |
| MM1.5 3BModel Parameters=3B, Decoding Strategy=Greedy2024.09 | 56.3 | — | — | — | — | — | — | |
| MM1.5-1B-MoEModel Size Class=1B, Shot Count=4-shot2024.09 | 56 | 33 | 56.5 | 89 | 56.7 | 56.7 | 44 | |
| MM1.5-7BModel Size Class=7B, Shot Count=4-shot2024.09 | 56 | 25.5 | 52.8 | 98.5 | 68.3 | 60 | 31 | |
| MM1.5 1B (MoE)Model Parameters=1B, Architecture=MoE, Decoding Strategy=Greedy2024.09 | 56 | — | — | — | — | — | — | |
| MM1-7BModel Size Class=7B, Shot Count=4-shot2024.09 | 52.8 | 33 | 69.5 | 97.5 | 40 | 45 | 32 | |
| MM1-30BModel Size Class=Larger (>30B), Shot Count=4-shot2024.09 | 52.1 | 25 | 63 | 98.5 | 51.7 | 38.3 | 36 | |
| MM1.5-1BModel Size Class=1B, Shot Count=4-shot2024.09 | 51 | 39 | 52 | 84 | 60 | 36.7 | 34 | |
| MM1.5 1BModel Parameters=1B, Decoding Strategy=Greedy2024.09 | 51 | — | — | — | — | — | — | |
| MM1-3BModel Size Class=3B, Shot Count=4-shot2024.09 | 37 | 27.5 | 50 | 79 | 18.3 | 13.3 | 34 | |
| Idefics-80BModel Size Class=Larger (>30B), Shot Count=4-shot2024.09 | 35.6 | 31.5 | 50 | 52.5 | 21.7 | 28.3 | 29.5 | |
| MM1-1BModel Size Class=1B, Shot Count=4-shot2024.09 | 34.3 | 25 | 49.3 | 73 | 16.7 | 8.3 | 33.5 | |
| Qwen-VL-Chat-7BModel Size Class=7B, Shot Count=4-shot2024.09 | 31.9 | 26.8 | 56.4 | 58 | 18.9 | 8.9 | 22.3 | |
| InternLM-XComposer2-7BModel Size Class=7B, Shot Count=4-shot2024.09 | 30.9 | 20 | 50.1 | 49 | 39.4 | 11.1 | 16 | |
| Idefics-9BModel Size Class=7B, Shot Count=4-shot2024.09 | 28 | 27.7 | 50 | 53.8 | 7.8 | 6.1 | 22.8 | |
| Emu2-Chat-37BModel Size Class=Larger (>30B), Shot Count=4-shot2024.09 | 26.9 | 14.8 | 50 | 28.2 | 21.7 | 10 | 36.5 | |
| OpenFlamingo-9BModel Size Class=7B, Shot Count=4-shot2024.09 | 20.9 | 18.8 | 50 | 51.2 | 2.8 | 2.8 | 0 | |
| Phi-3-Vision-4BModel Size Class=3B, Shot Count=4-shot2024.09 | 19.5 | 17 | 50 | 1 | 26.7 | 8.3 | 14 | |
| Phi-3-Vision 4BModel Parameters=4B, Decoding Strategy=Greedy2024.09 | 19.5 | — | — | — | — | — | — | |
| InternVL2 2BModel Parameters=2B, Decoding Strategy=Beam Search2024.09 | 18.5 | — | — | — | — | — | — | |
| Otter-9BModel Size Class=7B, Shot Count=4-shot2024.09 | 17.9 | 8.2 | 50.4 | 28.5 | 12.2 | 7.2 | 0.8 | |
| LLaVA-NeXT-7BModel Size Class=7B, Shot Count=4-shot2024.09 | 12.7 | 17.8 | 50 | 0 | 3.3 | 5 | 0 |