Multi-task Language Understanding on MMLU (Vanilla vs Attack Performance)
83.01MMLU Accuracy (Vanilla)AutoRAS
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AutoRASEvaluation Protocol=Vanilla, Cost ($)=0.672026.06 | 83.01 | — | |
| Magentic-OneEvaluation Protocol=Vanilla, Cost ($)=0.272026.06 | 81.04 | — | |
| OWLEvaluation Protocol=Vanilla, Cost ($)=0.652026.06 | 78.43 | — | |
| CAMELEvaluation Protocol=Vanilla, Cost ($)=0.232026.06 | 69.93 | — | |
| AutoRASEvaluation Protocol=Attack, Cost ($)=0.672026.06 | — | 82.35 | |
| CAMELEvaluation Protocol=Attack, Cost ($)=0.522026.06 | — | 62.75 | |
| Magentic-OneEvaluation Protocol=Attack, Cost ($)=0.382026.06 | — | 68.62 | |
| OWLEvaluation Protocol=Attack, Cost ($)=0.672026.06 | — | 68.63 |