Open-ended MCQA on MMLU (16 subjects filtered)
97.11ASR@30REALISTA
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| REALISTATarget LLM=Llama-3-3B2026.05 | 97.11 | 0.91 | 2.16 | 0.86 | |
| REALISTATarget LLM=Llama-3-8B2026.05 | 93.6 | 1.33 | 1.15 | 3.48 | |
| ICDTarget LLM=Llama-3-3B2026.05 | 90.77 | 1.57 | 13.97 | 100 | |
| ICDTarget LLM=Llama-3-8B2026.05 | 87.32 | 1.79 | 13.97 | 100 | |
| LARGOTarget LLM=Llama-3-3B2026.05 | 84.71 | 1.93 | 41.09 | 97.42 | |
| SECATarget LLM=Llama-3-8B2026.05 | 82.97 | 1.72 | 0.29 | 2.59 | |
| SECATarget LLM=Llama-3-3B2026.05 | 79.61 | 1.75 | 0.72 | 0.87 | |
| LARGOTarget LLM=Llama-3-8B2026.05 | 57.92 | 2.68 | 47.11 | 96.45 | |
| Raw promptingTarget LLM=Llama-3-8B2026.05 | 54.4 | 2.04 | 1.58 | 0 | |
| Raw promptingTarget LLM=Llama-3-3B2026.05 | 45.48 | 1.92 | 1.58 | 0 | |
| REALISTATarget LLM=Qwen-2.5-7B2026.05 | 41.61 | 2.75 | 3.32 | 2.88 | |
| SECATarget LLM=Qwen-2.5-7B2026.05 | 32.47 | 2.32 | 1.15 | 3.76 | |
| SECATarget LLM=Qwen-2.5-14B2026.05 | 27.51 | 2.26 | 0.58 | 3.48 | |
| REALISTATarget LLM=Qwen-2.5-14B2026.05 | 27.24 | 2.49 | 0.87 | 3.17 | |
| LARGOTarget LLM=Qwen-2.5-7B2026.05 | 23.89 | 2.29 | 43.37 | 96.53 | |
| ICDTarget LLM=Qwen-2.5-14B2026.05 | 13.56 | 1.84 | 13.97 | 100 | |
| ICDTarget LLM=Qwen-2.5-7B2026.05 | 11.5 | 1.72 | 13.97 | 100 | |
| LARGOTarget LLM=Qwen-2.5-14B2026.05 | 8.95 | 1.51 | 38.31 | 97.4 | |
| Raw promptingTarget LLM=Qwen-2.5-7B2026.05 | 6.4 | 0.99 | 1.58 | 0 | |
| Raw promptingTarget LLM=Qwen-2.5-14B2026.05 | 1.62 | 0.5 | 1.58 | 0 |