Name and Type Extraction on BPMN dataset
77.1PrecisionSketch2Process
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Sketch2ProcessPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 77.1 | 61.7 | 67.8 | |
| Sketch2ProcessPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 76.6 | 61.2 | 67.4 | |
| GPT-4.1Pipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 73.3 | 71.6 | 72.1 | |
| GPT-4.1-miniPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 72 | 74.3 | 72.9 | |
| GPT-4oPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 71.9 | 69.5 | 70.4 | |
| GPT-4.1Pipeline=Only VLM, Evaluation Setting=Strict2025.11 | 71.8 | 70.2 | 70.7 | |
| Mistral-small-3.1Pipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 71.2 | 70.5 | 70.6 | |
| GPT-4.1-miniPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 70.6 | 72.9 | 71.5 | |
| GPT-4oPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 69.9 | 67.6 | 68.5 | |
| Mistral-small-3.1Pipeline=Only VLM, Evaluation Setting=Strict2025.11 | 69.6 | 69 | 69.1 | |
| GPT-4o-miniPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 68.3 | 55.5 | 60.6 | |
| GPT-4o-miniPipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 67.7 | 59.3 | 62.4 | |
| GPT-4o-miniPipeline=VLM + RapidOCR, Evaluation Setting=Relaxed2025.11 | 67.7 | 59.3 | 62.4 | |
| GPT-4o-miniPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 66.9 | 54.3 | 59.4 | |
| GPT-4o-miniPipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 66.5 | 58.3 | 61.3 | |
| GPT-4o-miniPipeline=VLM + RapidOCR, Evaluation Setting=Strict2025.11 | 66.5 | 58.3 | 61.3 | |
| GPT-4oPipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 66.3 | 70.7 | 67.9 | |
| GPT-4oPipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 65.9 | 71.2 | 67.9 | |
| GPT-4oPipeline=VLM + Tesseract, Evaluation Setting=Strict2025.11 | 65.3 | 69.8 | 67 | |
| GPT-4.1-nanoPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 64.7 | 49.8 | 55.1 | |
| GPT-4oPipeline=VLM + RapidOCR, Evaluation Setting=Relaxed2025.11 | 64.6 | 70.5 | 67 | |
| GPT-4oPipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 64.3 | 69.5 | 66.3 | |
| GPT-4o-miniPipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 64.3 | 57.9 | 60.2 | |
| GPT-4.1-nanoPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 63.8 | 49.2 | 54.4 | |
| Mistral-small-3.1Pipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 63.5 | 70.2 | 66.2 | |
| GPT-4.1-miniPipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 63.3 | 74.2 | 67.9 | |
| Mistral-small-3.1Pipeline=VLM + RapidOCR, Evaluation Setting=Relaxed2025.11 | 63.3 | 71 | 66.6 | |
| GPT-4o-miniPipeline=VLM + Tesseract, Evaluation Setting=Strict2025.11 | 63.1 | 57 | 59.1 | |
| GPT-4.1Pipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 62.9 | 72.7 | 67 | |
| GPT-4oPipeline=VLM + RapidOCR, Evaluation Setting=Strict2025.11 | 62.8 | 68.8 | 65.2 | |
| GPT-4.1Pipeline=VLM + RapidOCR, Evaluation Setting=Relaxed2025.11 | 62.4 | 71.6 | 66.3 | |
| Mistral-small-3.1Pipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 62.4 | 68.9 | 65.1 | |
| GPT-4.1-miniPipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 62 | 73.6 | 66.9 | |
| GPT-4.1-miniPipeline=VLM + RapidOCR, Evaluation Setting=Relaxed2025.11 | 61.8 | 73.5 | 66.7 | |
| GPT-4.1-miniPipeline=VLM + Tesseract, Evaluation Setting=Strict2025.11 | 61.8 | 72.6 | 66.4 | |
| Mistral-small-3.1Pipeline=VLM + Tesseract, Evaluation Setting=Strict2025.11 | 61.4 | 68.1 | 64.2 | |
| Mistral-small-3.1Pipeline=VLM + RapidOCR, Evaluation Setting=Strict2025.11 | 61.3 | 68.9 | 64.5 | |
| GPT-4.1Pipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 61.2 | 70.9 | 65.2 | |
| GPT-4.1-miniPipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 60.8 | 72.2 | 65.7 | |
| GPT-4.1-miniPipeline=VLM + RapidOCR, Evaluation Setting=Strict2025.11 | 60.7 | 72.2 | 65.5 | |
| GPT-4.1Pipeline=VLM + RapidOCR, Evaluation Setting=Strict2025.11 | 60.4 | 69.4 | 64.2 | |
| Mistral-small-3.1Pipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 60.4 | 66.9 | 63.1 | |
| GPT-4.1Pipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 59.3 | 68.8 | 63.3 | |
| GPT-4.1Pipeline=VLM + Tesseract, Evaluation Setting=Strict2025.11 | 58 | 67.4 | 61.9 | |
| GPT-4.1-nanoPipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 56.2 | 48.8 | 51.1 | |
| GPT-4.1-nanoPipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 55.6 | 48.2 | 50.6 | |
| GPT-4.1-nanoPipeline=VLM + RapidOCR, Evaluation Setting=Relaxed2025.11 | 55.2 | 46.8 | 49.7 | |
| GPT-4.1-nanoPipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 54.5 | 46.3 | 48.9 | |
| GPT-4.1-nanoPipeline=VLM + RapidOCR, Evaluation Setting=Strict2025.11 | 54.1 | 45.8 | 48.6 | |
| GPT-4.1-nanoPipeline=VLM + Tesseract, Evaluation Setting=Strict2025.11 | 53.7 | 45.6 | 48.1 | |
| Gemma-12BPipeline=VLM + RapidOCR, Evaluation Setting=Relaxed2025.11 | 52.4 | 47.2 | 48.8 | |
| Gemma-12BPipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 52.4 | 47.2 | 48.8 | |
| BPMN RedrawerPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 51.8 | 39.9 | 44.8 | |
| BPMN RedrawerPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 51.8 | 39.9 | 44.8 | |
| Gemma-12BPipeline=VLM + RapidOCR, Evaluation Setting=Strict2025.11 | 51.5 | 46.5 | 48.1 | |
| Gemma-12BPipeline=VLM + Tesseract, Evaluation Setting=Strict2025.11 | 51.5 | 46.5 | 48.1 | |
| Gemma-12BPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 51.2 | 38.3 | 43 | |
| Gemma-12BPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 50.2 | 37.7 | 42.2 | |
| Qwen-7BPipeline=VLM + RapidOCR, Evaluation Setting=Relaxed2025.11 | 49.3 | 63.9 | 54.9 | |
| Qwen-7BPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 48.7 | 57.8 | 52.4 | |
| Qwen-7BPipeline=VLM + RapidOCR, Evaluation Setting=Strict2025.11 | 48.2 | 62.2 | 53.6 | |
| Qwen-7BPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 48 | 56.9 | 51.7 | |
| Qwen-7BPipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 47.8 | 61.9 | 53.2 | |
| Qwen-7BPipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 47.7 | 59.7 | 52.4 | |
| Qwen-7BPipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 47.1 | 59 | 51.7 | |
| Qwen-7BPipeline=VLM + Tesseract, Evaluation Setting=Strict2025.11 | 46.7 | 60.3 | 51.9 | |
| Gemma-12BPipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 44.3 | 38.3 | 40.3 | |
| Gemma-4BPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 44 | 46.3 | 44.3 | |
| Gemma-4BPipeline=VLM + RapidOCR, Evaluation Setting=Relaxed2025.11 | 43.9 | 47.5 | 44.6 | |
| Gemma-4BPipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 43.9 | 47.5 | 44.6 | |
| Gemma-12BPipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 43.5 | 37.7 | 39.6 | |
| Gemma-4BPipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 43.5 | 46.3 | 44 | |
| Gemma-4BPipeline=VLM + RapidOCR, Evaluation Setting=Strict2025.11 | 42.9 | 46.5 | 43.6 | |
| Gemma-4BPipeline=VLM + Tesseract, Evaluation Setting=Strict2025.11 | 42.9 | 46.5 | 43.6 | |
| Gemma-4BPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 42.5 | 44.9 | 42.8 | |
| Gemma-4BPipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 42 | 44.9 | 42.5 | |
| Qwen-3BPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 31.4 | 32.9 | 31.1 | |
| Qwen-3BPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 31.1 | 32.5 | 30.8 | |
| LLaMA-3.2-11BPipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 30.4 | 25.8 | 27 | |
| LLaMA-3.2-11BPipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 30.3 | 25.7 | 26.9 | |
| Qwen-3BPipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 30 | 29.9 | 29 | |
| Qwen-3BPipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 29.1 | 29.2 | 28.2 | |
| LLaMA-3.2-11BPipeline=VLM + RapidOCR, Evaluation Setting=Relaxed2025.11 | 28.6 | 25.3 | 25.9 | |
| LLaMA-3.2-11BPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 28.2 | 26.1 | 26.5 | |
| LLaMA-3.2-11BPipeline=VLM + RapidOCR, Evaluation Setting=Strict2025.11 | 28.2 | 25 | 25.6 | |
| Pixtral-12BPipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 28.2 | 23.8 | 24.6 | |
| LLaMA-3.2-11BPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 28 | 25.9 | 26.3 | |
| LLaMA-3.2-11BPipeline=VLM + Tesseract, Evaluation Setting=Strict2025.11 | 27.5 | 26 | 26.1 | |
| LLaMA-3.2-11BPipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 27.5 | 26 | 26.1 | |
| Pixtral-12BPipeline=Only VLM, Evaluation Setting=Strict2025.11 | 27.4 | 23.1 | 23.8 | |
| Pixtral-12BPipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 25.3 | 25.5 | 24.8 | |
| Pixtral-12BPipeline=VLM + Tesseract, Evaluation Setting=Strict2025.11 | 25.2 | 25.4 | 24.7 | |
| Pixtral-12BPipeline=VLM + RapidOCR, Evaluation Setting=Relaxed2025.11 | 24.5 | 25.2 | 24.1 | |
| Pixtral-12BPipeline=VLM + Pix2Struct, Evaluation Setting=Relaxed2025.11 | 24.1 | 24.2 | 23.3 | |
| Pixtral-12BPipeline=VLM + RapidOCR, Evaluation Setting=Strict2025.11 | 23.9 | 24.5 | 23.5 | |
| Pixtral-12BPipeline=VLM + Pix2Struct, Evaluation Setting=Strict2025.11 | 23.8 | 23.8 | 23 | |
| Pixtral-largePipeline=Only VLM, Evaluation Setting=Relaxed2025.11 | 22.3 | 19.9 | 20.7 | |
| Pixtral-largePipeline=Only VLM, Evaluation Setting=Strict2025.11 | 22 | 19.6 | 20.5 | |
| Pixtral-largePipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 16.4 | 18.1 | 17 | |
| Qwen-3BPipeline=VLM + Tesseract, Evaluation Setting=Relaxed2025.11 | 16.4 | 13.8 | 14.3 |