Code Generation on HumanEval 164 problems (test)
0.009Cost ($)Proactive Shepherding
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Proactive ShepherdingStrategy=Routing, Training Domain=GSM8K, Fine-tuned on Code=false2026.01 | 0.009 | 62.8 | 62.7 | 1.03 | |
| Reactive ShepherdingStrategy=Cascading, Training Domain=GSM8K, Fine-tuned on Code=false2026.01 | 0.013 | 76.2 | 44.3 | 1.42 | |
| Oracle ShepherdingStrategy=Oracle, Training Domain=GSM8K, Fine-tuned on Code=false2026.01 | 0.014 | 83.5 | 38.6 | 1.63 | |
| RouteLLMStrategy=Routing, Training Domain=GSM8K, Fine-tuned on Code=false2026.01 | 0.017 | 76.2 | 26.3 | 1.07 | |
| ABCStrategy=Cascading, Training Domain=GSM8K, Fine-tuned on Code=false2026.01 | 0.019 | 76.2 | 15.8 | 0.94 | |
| GraphRouterStrategy=Routing, Training Domain=GSM8K, Fine-tuned on Code=false2026.01 | 0.02 | 78 | 14 | 0.98 | |
| FrugalGPTStrategy=Cascading, Training Domain=GSM8K, Fine-tuned on Code=false2026.01 | 0.021 | 82.9 | 7 | 1.08 |