Tool-use Agent Performance on GeoDisaster
98.75Tool Usage Success Rate (Any)MAS+SFT
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MAS+SFTType=Multi2026.06 | 98.75 | 98.26 | 98.41 | 92.46 | 82.43 | 88.16 | 94.69 | 99.43 | 99.58 | 99.58 | 99.58 | 91.01 | |
| MAS+SFT+GRPOType=Multi2026.06 | 98.56 | 97.76 | 98.72 | 94.24 | 90.11 | 88.88 | 95.89 | 100 | 100 | 100 | 100 | 91.59 | |
| o4-miniType=Single2026.06 | 23.26 | 20.93 | 23.26 | 29.12 | 26.64 | 30.59 | 44.19 | 44.19 | 86.97 | 86.97 | 86.97 | 47.1 | |
| GPT-5.5Type=Single2026.06 | 20.93 | 20.93 | 30.23 | 55.06 | 61.99 | 71.44 | 100 | 100 | 95.86 | 95.86 | 95.86 | 82.32 | |
| GPT-5Type=Single2026.06 | 11.99 | 11.99 | 13.7 | 46.27 | 60.48 | 65.37 | 81.51 | 81.51 | 69.71 | 71.56 | 69.71 | 69.27 | |
| MASType=Multi2026.06 | 6.85 | 4.11 | 10.62 | 20.36 | 22.93 | 18.26 | 49.66 | 90.88 | 81.71 | 95.46 | 62.57 | 30.03 | |
| GPT-4oType=Single2026.06 | 3.44 | 3.44 | 3.78 | 42.65 | 60.19 | 65.43 | 80.76 | 80.76 | 78.53 | 78.53 | 78.53 | 71.23 | |
| Qwen2.5-7B-Inst.Type=Single2026.06 | 2.4 | 0.34 | 4.45 | 3.11 | 2.53 | 4.45 | 5.82 | 5.82 | 42.06 | 77.05 | 36.52 | 6.45 | |
| Qwen3-4B-Inst.Type=Single2026.06 | 0 | 0 | 0 | 1.16 | 1.16 | 2.33 | 2.33 | 6.98 | 40 | 23.26 | 9.3 | 3.72 | |
| Llama-3.1-8B-Inst.Type=Single2026.06 | 0 | 0 | 0 | 0.15 | 0.25 | 0.25 | 0.25 | 0.25 | 0 | 0.7 | 0.35 | 0.35 | |
| Mistral-7B-Inst.Type=Single2026.06 | 0 | 0 | 0 | 0.15 | 0.25 | 0.25 | 0.25 | 0.25 | 0 | 0.7 | 0.35 | 0.35 | |
| Qwen2.5-VL-7BType=Single2026.06 | 0 | 0 | 0 | 0.95 | 1.5 | 2.1 | 1.16 | 3.49 | 0 | 12.4 | 5.81 | 3.1 |