Diagram-to-Code on DiagramGen (test)
68.89Pass@1DiagramAgent
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| DiagramAgentSize=7B, configuration=Full model2024.11 | 68.89 | 48.99 | 84.64 | 72.74 | 46.98 | 37.46 | |
| GPT-4oSize=Not Specified2024.11 | 64.07 | 39.95 | 81.78 | 86.68 | 34.4 | 26.18 | |
| DiagramAgentSize=7B, configuration=w/o GPT-4o2024.11 | 62.59 | 48.71 | 84.57 | 72.69 | 46.49 | 37.22 | |
| DiagramAgentSize=7B, configuration=w/o Compiler2024.11 | 53.33 | 48.46 | 84.52 | 73.51 | 46.91 | 36.66 | |
| DiagramAgentSize=7B, configuration=w/o GPT-4o & Compiler2024.11 | 52.59 | 47.81 | 84.21 | 74.56 | 45.28 | 35.81 | |
| GLM-4-plusSize=Not Specified2024.11 | 51.48 | 35.92 | 80.16 | 86.12 | 29.1 | 24.6 | |
| Deepseek-vlSize=7B2024.11 | 50.74 | 25.18 | 76.48 | 88.82 | 18.35 | 16.13 | |
| Qwen2-VLSize=8B2024.11 | 28.89 | 31.74 | 80.04 | 88.13 | 28.39 | 21.21 | |
| Llama-3.2-VisionSize=11B2024.11 | 27.78 | 21.94 | 75.37 | 92.92 | 16.37 | 13.95 | |
| Phi-3.5-visionSize=4B2024.11 | 24.07 | 27.53 | 76.56 | 90.01 | 20.86 | 17.96 | |
| Gemini-1.5-proSize=Not Specified2024.11 | 17.78 | 38.66 | 80.75 | 88.05 | 30 | 25.62 | |
| Llava-v1.6Size=34B2024.11 | 8.89 | 26.68 | 76.53 | 93.46 | 21 | 16.3 | |
| Cogvlm2-llama3Size=19B2024.11 | 3.7 | 14.42 | 70.72 | 97.07 | 8.27 | 8.91 | |
| Internlm-xcomposer2.5Size=7B2024.11 | 3.33 | 28.47 | 77.35 | 92.35 | 18.74 | 17.97 | |
| Yi-VLSize=34B2024.11 | 2.22 | 20.01 | 70.57 | 95.43 | 11.68 | 12.53 |