Multiple Choice Question Answering on MCQA
93.5AccuracyRouteLLM 4o
Evaluation results
| Method | Links | ||
|---|---|---|---|
| RouteLLM 4oAPI Cost ($)=51.6, Time (minutes)=151.992025.05 | 93.5 | — | |
| o1 singleAPI Cost ($)=59.52, Time (minutes)=168.682025.05 | 93.5 | — | |
| RouteLLM 4o-miniAPI Cost ($)=53.14, Time (minutes)=162.282025.05 | 92.9 | — | |
| RouteLLM o1-miniAPI Cost ($)=51.27, Time (minutes)=152.142025.05 | 92.7 | — | |
| LLM-ATAPI Cost ($)=16.89, Time (minutes)=93.692025.05 | 90.6 | — | |
| Reptile-MetaICLModel=QWEN2, ICL setting=Yes2025.08 | 82.2 | 0.212 | |
| ABMLL-MetaICLModel=QWEN2, ICL setting=Yes2025.08 | 82.1 | 0.217 | |
| MetaICLModel=QWEN2, ICL setting=Yes2025.08 | 81.3 | 0.213 | |
| Reptile-MetaICLModel=LLAMA3, ICL setting=Yes2025.08 | 80.2 | 0.246 | |
| ABMLL-MetaICLModel=LLAMA3, ICL setting=Yes2025.08 | 80.2 | 0.244 | |
| MetaICLModel=LLAMA3, ICL setting=Yes2025.08 | 80.1 | 0.251 | |
| T5 (beam)QA Model=T52026.04 | 56.76 | — | |
| TinyLlamaQA Model=T52026.04 | 50.19 | — | |
| Ground-TruthQA Model=T52026.04 | 48.26 | — | |
| T5 (contrast)QA Model=T52026.04 | 47.1 | — | |
| GPT-3 (clustering)QA Model=T52026.04 | 42.08 | — | |
| GPT-3 (k-NN)QA Model=T52026.04 | 41.31 | — | |
| T5 (beam)QA Model=BART2026.04 | 40.12 | — | |
| Full-sequence DiffuMamba-HLR=4×10−32026.07 | 37.32 | — | |
| BDLM attentionLR=4×10−32026.07 | 36.31 | — | |
| GPT-3 (COT-k-NN)QA Model=T52026.04 | 36.29 | — | |
| Partially Reverse BDLM Mamba-HLR=8×10−32026.07 | 35.58 | — | |
| T5 (contrast)QA Model=BART2026.04 | 35.14 | — | |
| Full-sequence attentionLR=8×10−32026.07 | 35.03 | — | |
| Ground-TruthQA Model=BART2026.04 | 30.12 | — | |
| TinyLlamaQA Model=BART2026.04 | 26.64 | — | |
| GPT-3 (clustering)QA Model=BART2026.04 | 26.25 | — | |
| GPT-3 (k-NN)QA Model=BART2026.04 | 25.87 | — | |
| GPT-3 (COT-k-NN)QA Model=BART2026.04 | 22.78 | — |