Multiple-Choice Question Answering on MM-Telco
84.5CT WG1 AccuracyGPT-4o
Evaluation results
| Method | Links | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4o2025.11 | 84.5 | 88.7 | 85.9 | 84.7 | 79.3 | 87.3 | 88.3 | 70.9 | 76.4 | 89.1 | 87.6 | 85.5 | 88.7 | 90.1 | 89.3 | 85.6 | |
| Llama-VL-Telco2025.11 | 82.8 | 88.7 | 83.1 | 86.6 | 86.8 | 83.6 | 87.9 | 75.1 | 79.7 | 87.1 | 87.2 | 84.8 | 86.7 | 88.3 | 88.2 | 84.9 | |
| Phi 42025.11 | 81 | 87.6 | 83.6 | 83.1 | 76.7 | 83.9 | 86.7 | 66.7 | 78.2 | 86.4 | 85.7 | 83.6 | 87.7 | 88.2 | 89.1 | 83.7 | |
| Nemotron 70B2025.11 | 80.8 | 84.8 | 81.1 | 79.3 | 77.4 | 82.5 | 83.1 | 69.1 | 71.9 | 86.6 | 84.6 | 82.3 | 83.6 | 88 | 86.2 | 81.8 | |
| Llama 3.3 70B2025.11 | 80.6 | 85.7 | 83.6 | 80.5 | 75.4 | 82 | 83.5 | 70.3 | 73 | 86.8 | 84.6 | 81.5 | 83.4 | 86.2 | 85.4 | 81.9 | |
| Qwen2.5 72B2025.11 | 79.9 | 86.4 | 81.8 | 79.5 | 78 | 84 | 85.3 | 68.9 | 74.8 | 85.7 | 85.1 | 82.8 | 84.2 | 88.2 | 85.6 | 82.4 | |
| GPT-4o-mini2025.11 | 75.7 | 81.2 | 80.7 | 76.5 | 70.3 | 79.3 | 77.9 | 66 | 69.4 | 83.8 | 81.8 | 79 | 80.4 | 84.2 | 82.7 | 78.4 | |
| Llama 3.2 11B2025.11 | 69 | 78.9 | 71.1 | 69.7 | 66.4 | 71.2 | 75.8 | 57.8 | 64.8 | 78.3 | 77.1 | 73.9 | 75.8 | 75.3 | 79.3 | 72.6 | |
| Llama 3.2 3B2025.11 | 64 | 70.5 | 68.8 | 64.5 | 61.9 | 63.8 | 68.4 | 52.6 | 58.8 | 75.5 | 70.9 | 65.8 | 69.8 | 72.4 | 73.7 | 66.8 |