Coding Reasoning on Pancreas (test)
39.28Success RateCodeCytos + Few Shot
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CodeCytos + Few ShotBackbone=Qwen3-Coder-30B-A3B-Instruct, Framework=CodeAct-based, Prompting Strategy=Few Shot2026.05 | 39.28 | 60.18 | 64.86 | 64.76 | 57 | |
| CodeCytos + One ShotBackbone=Qwen3-Coder-30B-A3B-Instruct, Framework=CodeAct-based, Prompting Strategy=One Shot2026.05 | 33.67 | 58.5 | 62.79 | 62.93 | 55 | |
| CodeCytos + Few ShotBackbone=Qwen2.5-Coder-32B-Instruct, Framework=CodeAct-based, Prompting Strategy=Few Shot2026.05 | 32.44 | 57.22 | 60.32 | 60.41 | 53 | |
| CodeCytos + Few ShotBackbone=Facebook CWM, Framework=CodeAct-based, Prompting Strategy=Few Shot2026.05 | 27.1 | 53.18 | 54.69 | 54.7 | 49 | |
| CodeCytos + One ShotBackbone=Facebook CWM, Framework=CodeAct-based, Prompting Strategy=One Shot2026.05 | 23.65 | 50.49 | 57.16 | 57.33 | 46 | |
| CodeCytosBackbone=Qwen3-Coder-30B-A3B-Instruct, Framework=CodeAct-based, Prompting Strategy=Base2026.05 | 17.62 | 41.58 | 46.57 | 46.52 | 38 | |
| CodeCytosBackbone=Facebook CWM, Framework=CodeAct-based, Prompting Strategy=Base2026.05 | 14.49 | 43 | 51.02 | 51.03 | 38 | |
| Tool-Augmented LLM (Without CoT)Backbone=Qwen2.5-Coder-32B-Instruct, Framework=Conventional, Prompting Strategy=Without CoT2026.05 | 4.02 | 15.14 | 20.15 | 20.03 | 13 | |
| Tool-Augmented LLM (Zero-Shot CoT)Backbone=Qwen2.5-Coder-32B-Instruct, Framework=With, Prompting Strategy=Zero-Shot CoT2026.05 | 4.01 | 14.18 | 18.26 | 18.28 | 13 | |
| Tool-Augmented LLM (Few-Shot CoT)Backbone=Qwen2.5-Coder-32B-Instruct, Framework=With, Prompting Strategy=Few-Shot CoT2026.05 | 3.78 | 14.85 | 20.76 | 20.8 | 13 | |
| Tool-Augmented LLM (Without CoT)Backbone=Qwen3-Coder-30B-A3B-Instruct, Framework=Conventional, Prompting Strategy=Without CoT2026.05 | 1.26 | 5.64 | 8.14 | 8.17 | 5 | |
| Tool-Augmented LLM (Without CoT)Backbone=Facebook CWM, Framework=Conventional, Prompting Strategy=Without CoT2026.05 | 1.19 | 5.2 | 7.58 | 7.57 | 5 | |
| Tool-Augmented LLM (Few-Shot CoT)Backbone=Qwen3-Coder-30B-A3B-Instruct, Framework=With, Prompting Strategy=Few-Shot CoT2026.05 | 1.18 | 5.2 | 7.56 | 7.56 | 5 | |
| Tool-Augmented LLM (Zero-Shot CoT)Backbone=Facebook CWM, Framework=With, Prompting Strategy=Zero-Shot CoT2026.05 | 0.71 | 2.06 | 3.31 | 5.03 | 3 | |
| CodeCytos + One ShotBackbone=Qwen2.5-Coder-32B-Instruct, Framework=CodeAct-based, Prompting Strategy=One Shot2026.05 | 0.63 | 2.69 | 3.75 | 3.75 | 2 | |
| CodeCytosBackbone=Qwen2.5-Coder-32B-Instruct, Framework=CodeAct-based, Prompting Strategy=Base2026.05 | 0.62 | 3.16 | 5.04 | 5.01 | 3 | |
| Tool-Augmented LLM (Zero-Shot CoT)Backbone=Qwen3-Coder-30B-A3B-Instruct, Framework=With, Prompting Strategy=Zero-Shot CoT2026.05 | 0.5 | 2.23 | 3.16 | 3.16 | 2 | |
| Tool-Augmented LLM (Few-Shot CoT)Backbone=Facebook CWM, Framework=With, Prompting Strategy=Few-Shot CoT2026.05 | 0.39 | 1.18 | 1.97 | 3.14 | 2 |