Objective-C Type Recovery on MOTIF-Bench
94.7Avg AccuracyClaude Sonnet 4
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 4Access Type=API2026.01 | 94.7 | 86.7 | 72.3 | 77.4 | 0.1 | 100 | 0 | 100 | 83.2 | 100 | 95.5 | |
| Gemini-2.0 FlashAccess Type=API2026.01 | 88.7 | 82.9 | 60.8 | 66.8 | 6.5 | 100 | 0 | 100 | 79.2 | 100 | 75.5 | |
| Claude Sonnet 4evaluation_setup=One-Shot No-Tooling2026.01 | 78.7 | 80.1 | 62.8 | — | — | — | — | 99 | 65.9 | 73.8 | 76 | |
| ChatGPT 4oevaluation_setup=One-Shot No-Tooling2026.01 | 63 | 74.5 | 51.8 | — | — | — | — | 100 | 50.7 | 52.3 | 49 | |
| Gemini 2.0 flashevaluation_setup=One-Shot No-Tooling2026.01 | 57.3 | 71.5 | 48.6 | — | — | — | — | 100 | 39.6 | 43.1 | 46.6 | |
| XAI GrokAccess Type=API2026.01 | 56.4 | 82.1 | 73 | 99.7 | 5.6 | 100 | 0 | 82.1 | 50.7 | 48.1 | 44.6 | |
| Deepseek R1Access Type=API2026.01 | 54.8 | 79.5 | 61.1 | 99.9 | 53.7 | 97.7 | 2.3 | 96 | 42 | 35.8 | 45.5 | |
| GPT-4oAccess Type=API2026.01 | 50 | 86.3 | 68.9 | 99.9 | 14.4 | 100 | 0 | 99.4 | 44.3 | 17 | 39.2 | |
| Deepseek R1evaluation_setup=One-Shot No-Tooling2026.01 | 48.1 | 58.9 | 43.6 | — | — | — | — | 71.7 | 47.9 | 36.9 | 39.4 | |
| Qwen3 (8B) finetunedModel Size=8B, Finetuning Status=finetuned, Evaluation Setup=Task-Aligned Tool-Calling2026.01 | 42.4 | 83.5 | 64.7 | 99.5 | 16.3 | 100 | 0 | 99.4 | 32.2 | 9.6 | 28.6 | |
| gpt-oss-120bModel Size=120B, Access Type=Open-source2026.01 | 41.5 | 68.3 | 61.6 | 85.2 | 40.9 | 98.3 | 1.7 | 81.1 | 31.1 | 23.8 | 29.9 | |
| Qwen3 (14B)Model Size=14B, Access Type=Open-source2026.01 | 39.1 | 73.8 | 49.5 | 97.7 | 46.6 | 100 | 0 | 99.4 | 31.4 | 1.9 | 23.8 | |
| Llama 3.1 (8B-Instruct) finetunedModel Size=8B, Finetuning Status=finetuned, Evaluation Setup=Task-Aligned Tool-Calling2026.01 | 37.6 | 80.3 | 58.1 | 97.9 | 9.1 | 100 | 0 | 93.6 | 33 | 3.8 | 20.1 | |
| gpt-oss-20bModel Size=20B, Access Type=Open-source2026.01 | 28.5 | 57.1 | 40.2 | 82.4 | 30.4 | 82.6 | 17.4 | 68.8 | 16.7 | 0 | 28.6 | |
| Phi-3-mediumAccess Type=Open-source2026.01 | 26.8 | 46.1 | 22.7 | 0 | 22.6 | — | — | 82.2 | 9.1 | 3.8 | 11.9 | |
| Static Analysistype=Baseline2026.01 | 24.5 | 14.9 | 0 | — | — | — | — | 98 | 0 | 0 | 0 | |
| Llama 3.1 8B InstructModel Size=8B, Access Type=Open-source2026.01 | 19.4 | 35.9 | 19 | 91.5 | 10.3 | 100 | 0 | 58.5 | 7.6 | 2 | 9.4 |