Language Understanding on MMLU o=1 (Semantic-level)
76.9AccuracyW/O Decontamination
Evaluation results
| Method | Links | ||
|---|---|---|---|
| W/O DecontaminationModel=Qwen 2.5, Split=fcon, Access=-2026.01 | 76.9 | 0.098 | |
| ITDModel=Qwen 2.5, Split=fcon, Access=Black-box2026.01 | 76.6 | 0.095 | |
| TEDModel=LLaMA-3, Split=fcon, Access=Black-box2026.01 | 73.5 | 0.17 | |
| W/O DecontaminationModel=LLaMA-3, Split=fcon, Access=-2026.01 | 72.8 | 0.163 | |
| DeconIEPModel=Qwen 2.5, Split=fcon, Access=White-box2026.01 | 72.5 | 0.054 | |
| ITDModel=LLaMA-3, Split=fcon, Access=Black-box2026.01 | 68.3 | 0.118 | |
| Short CircuitModel=LLaMA-3, Split=fcon, Access=White-box2026.01 | 68.2 | 0.117 | |
| baselineModel=Qwen 2.5, Split=fclean, Access=-2026.01 | 67.1 | — | |
| TEDModel=Qwen 2.5, Split=fcon, Access=Black-box2026.01 | 63.2 | 0.039 | |
| Short CircuitModel=Mistral, Split=fcon, Access=White-box2026.01 | 62.3 | 0.189 | |
| Short CircuitModel=Qwen 2.5, Split=fcon, Access=White-box2026.01 | 62 | 0.051 | |
| W/O DecontaminationModel=Mistral, Split=fcon, Access=-2026.01 | 60.1 | 0.167 | |
| ITDModel=Mistral, Split=fcon, Access=Black-box2026.01 | 58.8 | 0.154 | |
| Shortcut NeuronModel=LLaMA-3, Split=fcon, Access=White-box2026.01 | 56.6 | 0.001 | |
| baselineModel=LLaMA-3, Split=fclean, Access=-2026.01 | 56.5 | — | |
| DeconIEPModel=LLaMA-3, Split=fcon, Access=White-box2026.01 | 52.5 | 0.041 | |
| DeconIEPModel=Mistral, Split=fcon, Access=White-box2026.01 | 46.7 | 0.033 | |
| baselineModel=Mistral, Split=fclean, Access=-2026.01 | 43.4 | — | |
| Shortcut NeuronModel=Qwen 2.5, Split=fcon, Access=White-box2026.01 | 41.7 | 0.254 | |
| TEDModel=Mistral, Split=fcon, Access=Black-box2026.01 | 40.5 | 0.029 | |
| Shortcut NeuronModel=Mistral, Split=fcon, Access=White-box2026.01 | 30.1 | 0.133 |