Diagram Understanding and Reasoning on DIAGRAMMA
59.64Accuracy (All)Claude-3.5-Sonnet
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Claude-3.5-Sonnet2026.03 | 59.64 | 64.59 | 42.98 | 74.69 | 53.33 | 77.27 | 54.55 | |
| GPT-4o2026.03 | 57.28 | 63.09 | 50.58 | 60.17 | 53.33 | 50 | 36.36 | |
| Qwen2-VL-72BParameter Scale=72B, Evaluation Platform=OpenRouter API2026.03 | 50.85 | 59.1 | 42.69 | 51.45 | 46.67 | 31.82 | 45.45 | |
| Claude3-Opus2026.03 | 49.15 | 54.11 | 40.35 | 55.6 | 33.33 | 45.45 | 50 | |
| Gemini-1.5-Flash2026.03 | 47.54 | 55.36 | 40.06 | 45.64 | 50 | 54.55 | 31.82 | |
| Claude3-Sonnet2026.03 | 47.54 | 50.62 | 38.01 | 58.92 | 46.67 | 36.36 | 27.27 | |
| LLama3.2-VL-90BParameter Scale=90B, Evaluation Platform=OpenRouter API2026.03 | 46.88 | 50.37 | 41.23 | 53.53 | 26.67 | 40.91 | 31.82 | |
| LLama3.2-VL-11BParameter Scale=11B2026.03 | 46.22 | 48.38 | 40.06 | 56.02 | 30 | 50 | 13.64 | |
| Pixtral-12bParameter Scale=12b, Evaluation Platform=OpenRouter API2026.03 | 44.71 | 50.62 | 40.06 | 44.81 | 43.33 | 36.36 | 18.18 | |
| GPT-4o-mini2026.03 | 44.42 | 47.63 | 36.55 | 53.53 | 26.67 | 45.45 | 31.82 | |
| Gemini-1.5-pro2026.03 | 44.23 | 49.13 | 41.23 | 42.74 | 30 | 59.09 | 22.73 | |
| LLava-OneVision-Qwen2-7bParameter Scale=7b2026.03 | 42.91 | 48.13 | 35.38 | 44.4 | 53.33 | 45.45 | 31.82 | |
| Claude3-Haiku2026.03 | 42.53 | 46.63 | 31.58 | 54.77 | 30 | 36.36 | 27.27 | |
| Qwen2-vl-7BParameter Scale=7B2026.03 | 42.16 | 48.63 | 35.38 | 40.66 | 43.33 | 45.45 | 40.91 | |
| InternVL2-8BParameter Scale=8B2026.03 | 41.02 | 47.38 | 35.09 | 39.83 | 40 | 40.91 | 31.82 | |
| Phi-VL-3.5Model Version=3.52026.03 | 38.19 | 42.64 | 33.04 | 40.66 | 30 | 40.91 | 18.18 | |
| Minicpm-2.6Model Version=2.62026.03 | 35.44 | 41.65 | 29.82 | 36.51 | 26.67 | 27.27 | 18.18 |