Tool-use decision-making on MetaTool (test)
90.2Decision AccuracyMeCo
Evaluation results
| Method | Links | |
|---|---|---|
| MeCoBackbone=Mist-7B, Fine-tuning=Post Fine-tuning, Context=w/ ctx2025.02 | 90.2 | |
| NaiveBackbone=Mist-7B, Fine-tuning=Post Fine-tuning, Context=w/ ctx2025.02 | 89.2 | |
| PyesBackbone=Mist-7B, Fine-tuning=Post Fine-tuning, Context=w/ ctx2025.02 | 89.2 | |
| MeCoBackbone=LM3-70B, Fine-tuning=Post Fine-tuning, Context=w/ ctx2025.02 | 87.3 | |
| MeCoBackbone=Mist-7B, Fine-tuning=Post Fine-tuning, Context=w/o ctx2025.02 | 86.5 | |
| PyesBackbone=LM3-70B, Fine-tuning=Post Fine-tuning, Context=w/ ctx2025.02 | 86.2 | |
| NaiveBackbone=LM3-70B, Fine-tuning=Post Fine-tuning, Context=w/ ctx2025.02 | 86 | |
| NaiveBackbone=Mist-7B, Fine-tuning=Post Fine-tuning, Context=w/o ctx2025.02 | 86 | |
| MeCoBackbone=LM3-70B, Fine-tuning=Pre Fine-tuning, Context=w/ ctx2025.02 | 85.4 | |
| PyesBackbone=Mist-7B, Fine-tuning=Post Fine-tuning, Context=w/o ctx2025.02 | 85 | |
| PyesBackbone=LM3-70B, Fine-tuning=Pre Fine-tuning, Context=w/ ctx2025.02 | 84.8 | |
| NaiveBackbone=LM3-70B, Fine-tuning=Pre Fine-tuning, Context=w/ ctx2025.02 | 84.6 | |
| GPT-4-turboStrategy=Naive, Fine-tuning=Pre Fine-tuning, Context=w/ ctx2025.02 | 84.4 | |
| MeCoBackbone=LM3-8B, Fine-tuning=Post Fine-tuning, Context=w/ ctx2025.02 | 84.3 | |
| MeCoBackbone=LM3-8B, Fine-tuning=Post Fine-tuning, Context=w/o ctx2025.02 | 82.3 | |
| NaiveBackbone=LM3-8B, Fine-tuning=Post Fine-tuning, Context=w/ ctx2025.02 | 82.1 | |
| PyesBackbone=LM3-8B, Fine-tuning=Post Fine-tuning, Context=w/ ctx2025.02 | 81.7 | |
| MeCoBackbone=LM3-70B, Fine-tuning=Post Fine-tuning, Context=w/o ctx2025.02 | 81.2 | |
| NaiveBackbone=LM3-8B, Fine-tuning=Post Fine-tuning, Context=w/o ctx2025.02 | 80.8 | |
| PyesBackbone=LM3-8B, Fine-tuning=Post Fine-tuning, Context=w/o ctx2025.02 | 80.8 | |
| MeCoBackbone=LM3-70B, Fine-tuning=Pre Fine-tuning, Context=w/o ctx2025.02 | 79.6 | |
| NaiveBackbone=LM3-70B, Fine-tuning=Post Fine-tuning, Context=w/o ctx2025.02 | 77.7 | |
| PyesBackbone=LM3-70B, Fine-tuning=Post Fine-tuning, Context=w/o ctx2025.02 | 77.1 | |
| MeCoBackbone=Mist-7B, Fine-tuning=Pre Fine-tuning, Context=w/ ctx2025.02 | 75.4 | |
| MeCoBackbone=Mist-7B, Fine-tuning=Pre Fine-tuning, Context=w/o ctx2025.02 | 74.7 | |
| MeCoBackbone=LM3-8B, Fine-tuning=Pre Fine-tuning, Context=w/o ctx2025.02 | 74 | |
| PyesBackbone=LM3-70B, Fine-tuning=Pre Fine-tuning, Context=w/o ctx2025.02 | 73.7 | |
| PyesBackbone=Mist-7B, Fine-tuning=Pre Fine-tuning, Context=w/o ctx2025.02 | 73.1 | |
| PyesBackbone=Mist-7B, Fine-tuning=Pre Fine-tuning, Context=w/ ctx2025.02 | 71.2 | |
| NaiveBackbone=Mist-7B, Fine-tuning=Pre Fine-tuning, Context=w/ ctx2025.02 | 69 | |
| NaiveBackbone=LM3-70B, Fine-tuning=Pre Fine-tuning, Context=w/o ctx2025.02 | 68.8 | |
| NaiveBackbone=Mist-7B, Fine-tuning=Pre Fine-tuning, Context=w/o ctx2025.02 | 68.5 | |
| MeCoBackbone=LM3-8B, Fine-tuning=Pre Fine-tuning, Context=w/ ctx2025.02 | 65 | |
| PyesBackbone=LM3-8B, Fine-tuning=Pre Fine-tuning, Context=w/ ctx2025.02 | 63.5 | |
| PyesBackbone=LM3-8B, Fine-tuning=Pre Fine-tuning, Context=w/o ctx2025.02 | 62.7 | |
| NaiveBackbone=LM3-8B, Fine-tuning=Pre Fine-tuning, Context=w/ ctx2025.02 | 61.9 | |
| GPT-4-turboStrategy=Naive, Fine-tuning=Pre Fine-tuning, Context=w/o ctx2025.02 | 61.3 | |
| NaiveBackbone=LM3-8B, Fine-tuning=Pre Fine-tuning, Context=w/o ctx2025.02 | 58.3 |