Relation and Type Extraction on BPMN
53.2PrecisionGPT-4.1-mini
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GPT-4.1-miniPipeline=Only VLM, Setting=Strict2025.11 | 53.2 | 48 | 46.9 | |
| GPT-4.1-miniPipeline=Only VLM, Setting=Relaxed2025.11 | 53.2 | 48 | 46.9 | |
| GPT-4.1Pipeline=Only VLM, Setting=Strict2025.11 | 51.9 | 47.4 | 47.5 | |
| GPT-4.1Pipeline=Only VLM, Setting=Relaxed2025.11 | 51.9 | 47.4 | 47.5 | |
| GPT-4oPipeline=Only VLM, Setting=Strict2025.11 | 42.4 | 31.7 | 33.6 | |
| GPT-4oPipeline=Only VLM, Setting=Relaxed2025.11 | 42.4 | 31.7 | 33.6 | |
| Mistral-small-3.1Pipeline=Only VLM, Setting=Strict2025.11 | 42.4 | 41.2 | 39.5 | |
| Mistral-small-3.1Pipeline=Only VLM, Setting=Relaxed2025.11 | 42.4 | 41.2 | 39.5 | |
| Pixtral-largePipeline=Only VLM, Setting=Strict2025.11 | 40.2 | 28.5 | 27.7 | |
| Pixtral-largePipeline=Only VLM, Setting=Relaxed2025.11 | 40.2 | 28.5 | 27.7 | |
| GPT-4.1-miniPipeline=VLM + Tesseract, Setting=Strict2025.11 | 33.7 | 47.1 | 34 | |
| GPT-4.1-miniPipeline=VLM + Tesseract, Setting=Relaxed2025.11 | 33.7 | 47.1 | 34 | |
| GPT-4.1Pipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 33.3 | 49.8 | 33.6 | |
| GPT-4.1Pipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 33.3 | 49.8 | 33.6 | |
| GPT-4.1-miniPipeline=VLM + RapidOCR, Setting=Strict2025.11 | 33.3 | 43.9 | 32.2 | |
| GPT-4.1-miniPipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 33.3 | 43.9 | 32.2 | |
| GPT-4.1-miniPipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 33 | 47.5 | 34 | |
| GPT-4.1-miniPipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 33 | 47.5 | 34 | |
| GPT-4.1Pipeline=VLM + Tesseract, Setting=Strict2025.11 | 32.1 | 42.6 | 30.7 | |
| GPT-4.1Pipeline=VLM + Tesseract, Setting=Relaxed2025.11 | 32.1 | 42.6 | 30.7 | |
| GPT-4.1Pipeline=VLM + RapidOCR, Setting=Strict2025.11 | 31.6 | 45.2 | 31.3 | |
| GPT-4.1Pipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 31.6 | 45.2 | 31.3 | |
| Mistral-small-3.1Pipeline=VLM + RapidOCR, Setting=Strict2025.11 | 26.2 | 39.5 | 29.5 | |
| Mistral-small-3.1Pipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 26.2 | 39.5 | 29.5 | |
| Mistral-small-3.1Pipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 26.1 | 42.1 | 29.9 | |
| Mistral-small-3.1Pipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 26.1 | 42.1 | 29.9 | |
| GPT-4oPipeline=VLM + RapidOCR, Setting=Strict2025.11 | 25.6 | 28 | 25.4 | |
| GPT-4oPipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 25.6 | 28 | 25.4 | |
| Qwen-7BPipeline=Only VLM, Setting=Strict2025.11 | 24.6 | 14.7 | 15.1 | |
| Qwen-7BPipeline=Only VLM, Setting=Relaxed2025.11 | 24.6 | 14.7 | 15.1 | |
| Pixtral-largePipeline=VLM + RapidOCR, Setting=Strict2025.11 | 24.5 | 13.7 | 13.2 | |
| Pixtral-largePipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 24.5 | 13.7 | 13.2 | |
| GPT-4oPipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 24.4 | 29 | 23.9 | |
| GPT-4oPipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 24.4 | 29 | 23.9 | |
| Pixtral-largePipeline=VLM + Tesseract, Setting=Strict2025.11 | 23.7 | 21.7 | 14.8 | |
| Pixtral-largePipeline=VLM + Tesseract, Setting=Relaxed2025.11 | 23.7 | 21.7 | 14.8 | |
| GPT-4o-miniPipeline=Only VLM, Setting=Strict2025.11 | 23.6 | 17.3 | 18.8 | |
| GPT-4o-miniPipeline=Only VLM, Setting=Relaxed2025.11 | 23.6 | 17.3 | 18.8 | |
| Mistral-small-3.1Pipeline=VLM + Tesseract, Setting=Strict2025.11 | 23.3 | 36.1 | 26.7 | |
| Mistral-small-3.1Pipeline=VLM + Tesseract, Setting=Relaxed2025.11 | 23.3 | 36.1 | 26.7 | |
| GPT-4.1-nanoPipeline=Only VLM, Setting=Strict2025.11 | 22.9 | 16.7 | 17.5 | |
| GPT-4.1-nanoPipeline=Only VLM, Setting=Relaxed2025.11 | 22.9 | 16.7 | 17.5 | |
| Qwen-7BPipeline=VLM + RapidOCR, Setting=Strict2025.11 | 22 | 14.5 | 15.2 | |
| Qwen-7BPipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 22 | 14.5 | 15.2 | |
| GPT-4oPipeline=VLM + Tesseract, Setting=Strict2025.11 | 21.4 | 28.4 | 22.2 | |
| GPT-4oPipeline=VLM + Tesseract, Setting=Relaxed2025.11 | 21.4 | 28.4 | 22.2 | |
| GPT-4o-miniPipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 20.7 | 18 | 17.5 | |
| GPT-4o-miniPipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 20.7 | 18 | 17.5 | |
| GPT-4o-miniPipeline=VLM + RapidOCR, Setting=Strict2025.11 | 20.7 | 18 | 17.5 | |
| GPT-4o-miniPipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 20.7 | 18 | 17.5 | |
| Sketch2ProcessPipeline=Only VLM, Setting=Strict2025.11 | 20.1 | 28.7 | 21.7 | |
| Sketch2ProcessPipeline=Only VLM, Setting=Relaxed2025.11 | 20.1 | 28.7 | 21.7 | |
| Qwen-7BPipeline=VLM + Tesseract, Setting=Strict2025.11 | 19.6 | 14.8 | 14.2 | |
| Qwen-7BPipeline=VLM + Tesseract, Setting=Relaxed2025.11 | 19.6 | 14.8 | 14.2 | |
| Qwen-7BPipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 19.5 | 13.6 | 14 | |
| Qwen-7BPipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 19.5 | 13.6 | 14 | |
| GPT-4o-miniPipeline=VLM + Tesseract, Setting=Strict2025.11 | 19.1 | 15.5 | 15.7 | |
| GPT-4o-miniPipeline=VLM + Tesseract, Setting=Relaxed2025.11 | 19.1 | 15.5 | 15.7 | |
| GPT-4.1-nanoPipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 18.7 | 19.1 | 17 | |
| GPT-4.1-nanoPipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 18.7 | 19.1 | 17 | |
| GPT-4.1-nanoPipeline=VLM + RapidOCR, Setting=Strict2025.11 | 18.4 | 16.4 | 16 | |
| GPT-4.1-nanoPipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 18.4 | 16.4 | 16 | |
| Gemma-12BPipeline=VLM + RapidOCR, Setting=Strict2025.11 | 18.2 | 15.9 | 15.8 | |
| Gemma-12BPipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 18.2 | 15.9 | 15.8 | |
| Gemma-12BPipeline=VLM + Tesseract, Setting=Strict2025.11 | 18.2 | 15.9 | 15.8 | |
| Gemma-12BPipeline=VLM + Tesseract, Setting=Relaxed2025.11 | 18.2 | 15.9 | 15.8 | |
| GPT-4.1-nanoPipeline=VLM + Tesseract, Setting=Strict2025.11 | 17.6 | 15.1 | 14.6 | |
| GPT-4.1-nanoPipeline=VLM + Tesseract, Setting=Relaxed2025.11 | 17.6 | 15.1 | 14.6 | |
| Pixtral-largePipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 17.3 | 23.2 | 15.1 | |
| Pixtral-largePipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 17.3 | 23.2 | 15.1 | |
| Gemma-12BPipeline=Only VLM, Setting=Strict2025.11 | 16.4 | 13.9 | 13.9 | |
| Gemma-12BPipeline=Only VLM, Setting=Relaxed2025.11 | 16.4 | 13.9 | 13.9 | |
| Gemma-4BPipeline=Only VLM, Setting=Strict2025.11 | 13 | 12.3 | 11.4 | |
| Gemma-4BPipeline=Only VLM, Setting=Relaxed2025.11 | 13 | 12.3 | 11.4 | |
| Gemma-4BPipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 12 | 11.8 | 10.7 | |
| Gemma-4BPipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 12 | 11.8 | 10.7 | |
| Gemma-12BPipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 11.7 | 12.3 | 10.9 | |
| Gemma-12BPipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 11.7 | 12.3 | 10.9 | |
| BPMN RedrawerPipeline=Only VLM, Setting=Strict2025.11 | 9.8 | 14.2 | 10.3 | |
| BPMN RedrawerPipeline=Only VLM, Setting=Relaxed2025.11 | 9.8 | 14.2 | 10.3 | |
| Gemma-4BPipeline=VLM + RapidOCR, Setting=Strict2025.11 | 9.6 | 8.8 | 8.2 | |
| Gemma-4BPipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 9.6 | 8.8 | 8.2 | |
| Gemma-4BPipeline=VLM + Tesseract, Setting=Strict2025.11 | 9.6 | 8.8 | 8.2 | |
| Gemma-4BPipeline=VLM + Tesseract, Setting=Relaxed2025.11 | 9.6 | 8.8 | 8.2 | |
| Qwen-3BPipeline=Only VLM, Setting=Strict2025.11 | 8.6 | 3.2 | 3.4 | |
| Qwen-3BPipeline=Only VLM, Setting=Relaxed2025.11 | 8.6 | 3.2 | 3.4 | |
| Pixtral-12BPipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 8.1 | 3.9 | 4.5 | |
| Pixtral-12BPipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 8.1 | 3.9 | 4.5 | |
| LLaMA-3.2-11BPipeline=VLM + RapidOCR, Setting=Strict2025.11 | 7.7 | 5.9 | 6 | |
| LLaMA-3.2-11BPipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 7.7 | 5.9 | 6 | |
| Pixtral-12BPipeline=VLM + RapidOCR, Setting=Strict2025.11 | 6.6 | 4.4 | 4.6 | |
| Pixtral-12BPipeline=VLM + RapidOCR, Setting=Relaxed2025.11 | 6.6 | 4.4 | 4.6 | |
| LLaMA-3.2-11BPipeline=Only VLM, Setting=Strict2025.11 | 5.8 | 5.4 | 5.1 | |
| LLaMA-3.2-11BPipeline=Only VLM, Setting=Relaxed2025.11 | 5.8 | 5.4 | 5.1 | |
| Qwen-3BPipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 5.8 | 2.9 | 2.8 | |
| Qwen-3BPipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 5.8 | 2.9 | 2.8 | |
| LLaMA-3.2-11BPipeline=VLM + Pix2Struct, Setting=Strict2025.11 | 5.7 | 3.8 | 4.2 | |
| LLaMA-3.2-11BPipeline=VLM + Pix2Struct, Setting=Relaxed2025.11 | 5.7 | 3.8 | 4.2 | |
| Pixtral-12BPipeline=VLM + Tesseract, Setting=Strict2025.11 | 5.3 | 3.9 | 4.1 | |
| Pixtral-12BPipeline=VLM + Tesseract, Setting=Relaxed2025.11 | 5.3 | 3.9 | 4.1 |