Algorithmic Program Execution on FP Algorithmic Task n=100 (test)
100AccuracyDeepSeek-Coder-33B-Instruct
Evaluation results
| Method | Links | |
|---|---|---|
| DeepSeek-Coder-33B-InstructProtocol=LLM-PV, Tools=no2025.10 | 100 | |
| OLMo-3-32B-InstructProtocol=LLM-PV, Tools=no2025.10 | 100 | |
| Gemini-3.1-ProProtocol=LLM-PV, Tools=with tools2025.10 | 100 | |
| GPT-5-ThinkingProtocol=LLM-PV, Tools=no2025.10 | 100 | |
| GPT-5-ThinkingProtocol=LLM-PV, Tools=with tools2025.10 | 100 | |
| GPT-5-ThinkingProtocol=ICL, Tools=with tools2025.10 | 99 | |
| GPT-5-ThinkingProtocol=ICL, Tools=no2025.10 | 82 | |
| MLAgentBench (GPT-5-Thinking)Protocol=Agentic, Tools=with tools2025.10 | 53.6 | |
| Qwen3-30B-InstructProtocol=ICL, Tools=no2025.10 | 53 | |
| DeepSeek-Coder-1.3BProtocol=SGD (From scratch)2025.10 | 52.2 | |
| Llama-3.2-1BProtocol=SGD (From scratch)2025.10 | 52.2 | |
| Qwen3-1.7BProtocol=Fine-tuning2025.10 | 51.2 | |
| DeepSeek-Coder-1.3BProtocol=Fine-tuning2025.10 | 50.5 | |
| Llama-3.2-1BProtocol=Fine-tuning2025.10 | 50.4 | |
| Qwen3-30B-InstructProtocol=LLM-PV, Tools=no2025.10 | 50.4 | |
| TabPFNProtocol=Pre-trained Tabular Model2025.10 | 50.1 | |
| Qwen3-Coder-30B-InstructProtocol=ICL, Tools=no2025.10 | 50 | |
| XGBoostProtocol=Classic ML2025.10 | 49.9 | |
| SVMProtocol=Classic ML2025.10 | 49.9 | |
| Random ForestProtocol=Classic ML2025.10 | 49.8 | |
| Genetic AlgorithmProtocol=Classic ML2025.10 | 49.8 | |
| AIDE-ML (GPT-5-Thinking)Protocol=Agentic, Tools=with tools2025.10 | 49.5 | |
| Qwen3-1.7BProtocol=SGD (From scratch)2025.10 | 49.3 | |
| Gemini-3.1-ProProtocol=ICL, Tools=with tools2025.10 | 48 | |
| DeepSeek-Coder-33B-InstructProtocol=ICL, Tools=no2025.10 | 43 |