Type Inference on Private macOS Frameworks (MOTIF) (test)
75.2PM AccuracyQwen3 (8B)
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3 (8B)Type=Task-Aligned Tool-Calling LLM, Training=finetuned, Rank=12026.01 | 75.2 | 48.4 | 100 | 0 | 100 | 0 | 60 | 40 | 0 | 33.3 | |
| Deepseek R1Framework=MOTIF, Access=API Access, Rank=22026.01 | 72.5 | 38.7 | 100 | 38.7 | 97.3 | 2.7 | 60 | 20 | 0 | 16.7 | |
| GPT-4oFramework=MOTIF, Access=API Access, Rank=32026.01 | 71.9 | 51.6 | 100 | 0 | 100 | 0 | 60 | 0 | 0 | 33.3 | |
| Gemini-2.0 FlashFramework=MOTIF, Access=API Access, Rank=42026.01 | 67.7 | 32.3 | 100 | 9.7 | 100 | 0 | 60 | 20 | 0 | 50 | |
| Claude Sonnet 4Framework=MOTIF, Access=API Access, Rank=52026.01 | 66.7 | 56.2 | 100 | 0 | 100 | 0 | 33.3 | 0 | 0 | 50 | |
| XAI Grok 4Framework=MOTIF, Access=API Access, Rank=62026.01 | 58.3 | 48.4 | 100 | 6.5 | 100 | 0 | 40 | 0 | 0 | 0 | |
| Phi-3-mediumFramework=MOTIF, Access=API Access, Rank=72026.01 | 37.3 | 6.5 | 0 | 6.5 | — | — | 60 | 0 | 0 | 0 | |
| Llama 3.1 8B InstructFramework=MOTIF, Access=API Access, Rank=82026.01 | 24.6 | 6.5 | 83.9 | 0 | 100 | 0 | 40 | 0 | 0 | 20 | |
| Static AnalysisType=Baseline, Rank=92026.01 | 22.3 | 0 | — | — | — | — | 100 | 0 | 0 | 0 |