Question Answering on ARC Challenge
96.3AccuracyGPT-4
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4Number of shots=25-shot2023.11 | 96.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Kimi-K2 Base# Shots=25-shot, # Activated Params=32B, # Total Params=1043B2026.01 | 96.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MiMo-V2-Flash Base# Shots=25-shot, # Activated Params=15B, # Total Params=309B2026.01 | 95.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepSeek-V3.1 Base# Shots=25-shot, # Activated Params=37B, # Total Params=671B2026.01 | 95.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepSeek-V3.2 Exp Base# Shots=25-shot, # Activated Params=37B, # Total Params=671B2026.01 | 95.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral Small 24B Inst 2501Classifier=Self-labeled2026.01 | 93.9 | — | — | 0 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral Small 24B Inst 2501Classifier=Majority-labeled2026.01 | 93.9 | — | — | 0 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 14B BaseClassifier=Self-labeled2026.01 | 93.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 14B BaseClassifier=Majority-labeled2026.01 | 93.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral Small 24B Base 2501Classifier=Self-labeled2026.01 | 92.9 | — | — | 0.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral Small 24B Base 2501Classifier=Majority-labeled2026.01 | 92.9 | — | — | 0.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 8B BaseClassifier=Self-labeled2026.01 | 92.5 | — | — | 0 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 8B BaseClassifier=Majority-labeled2026.01 | 92.5 | — | — | 0 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 8B BaseClassifier=Self-labeled2026.01 | 92.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 8B BaseClassifier=Majority-labeled2026.01 | 92.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 14BClassifier=Self-labeled2026.01 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 14BClassifier=Majority-labeled2026.01 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 8BClassifier=Self-labeled2026.01 | 91.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 8BClassifier=Majority-labeled2026.01 | 91.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KALEBackbone=Qwen2.5 32B, Category=Augmented-based2026.01 | 89.93 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SALAD-7BInput Modality=Text, Parameters=7B2025.10 | 89.2 | — | -0.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Cont-Squeezerank=128 to 1, additional_steps=2002026.02 | 89.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Direct Fine-tuningrank=1, additional_steps=7002026.02 | 89.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Direct Fine-tuningrank=1, additional_steps=02026.02 | 88.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| In-Squeezerank=128 to 1, schedule=Standard2026.02 | 88.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| In-Squeezerank=128 to 1, schedule=Min steps2026.02 | 88.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LinUCB+KLContextual=true2026.02 | 88.8 | — | — | 7.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Cont-Squeezerank=128 to 1, additional_steps=7002026.02 | 88.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Direct Fine-tuningrank=1, additional_steps=2002026.02 | 88.59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KALEBackbone=Gemma2 9B, Category=Augmented-based2026.01 | 88.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ours r=1Model=Qwen 3 4B, Params (%)=0.005%2026.02 | 88.4 | — | — | — | — | 0 | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 4B BaseClassifier=Self-labeled2026.01 | 88.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 4B BaseClassifier=Majority-labeled2026.01 | 88.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| JoLAModel=Qwen 3 4B, Params (%)=0.007%2026.02 | 88.1 | — | — | — | — | -0.3 | — | — | — | — | — | — | — | — | — | — | — | |
| FairSteerBackbone=Qwen-3-4B2026.01 | 87.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LFTFBackbone=Qwen-3-4B2026.01 | 87.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Falcon-180BNumber of shots=2-shot2023.11 | 87.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaseBackbone=Qwen-3-4B2026.01 | 87.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-Omni-7BInput Modality=Text, Parameters=7B2025.10 | 87.6 | — | 0.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoFITModel=Qwen 3 4B, Params (%)=0.003%2026.02 | 87.6 | — | — | — | — | -0.8 | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 4BClassifier=Self-labeled2026.01 | 87.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 4BClassifier=Majority-labeled2026.01 | 87.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CRISPRBackbone=Qwen-3-4B, Debias target=race2026.01 | 86.86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ours (1-vec)Model=Qwen 3 4B, Params (%)=0.003%2026.02 | 86.3 | — | — | — | — | -2.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Mixtral 8x7B25-shot=true2024.01 | 85.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CRISPRBackbone=Qwen-3-4B, Debias target=gender2026.01 | 85.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TASOModel=Qwen2.5 3B, #Param.=2.06 M, Evaluation Protocol=Zero-shot2025.09 | 85.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KnowBiasBackbone=Qwen-3-4B2026.01 | 85.49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT3MixBackbone=Qwen2.5 32B, Category=Augmented-based2026.01 | 85.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-3.5Number of shots=25-shot2023.11 | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-3.525-shot=true2024.01 | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaMA 2 70B25-shot=true2024.01 | 85.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GraphRAGBackbone=Gemma2 9B, Category=Retrieval-based2026.01 | 85.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AugGPTBackbone=Qwen2.5 32B, Category=Augmented-based2026.01 | 84.47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRA-XSModel=Qwen2.5 3B, #Param.=4.13 M, rank=128, Evaluation Protocol=Zero-shot2025.09 | 84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R1-OracleBackbone=Qwen2.5-3B-Instruct, Supervision Paradigm=100% GT2026.03 | 84 | — | 0 | — | — | — | 79.12 | — | — | — | — | — | — | — | — | — | — | |
| STaRBackbone=Qwen2.5 32B, Category=Augmented-based2026.01 | 83.87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| StructGPTBackbone=Gemma2 9B, Category=Retrieval-based2026.01 | 83.28 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DoRAModel=Qwen2.5 3B, #Param.=65.98 M, rank=32, Evaluation Protocol=Zero-shot2025.09 | 83.26 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SALAD-3BInput Modality=Text, Parameters=3B2025.10 | 83.1 | — | -1.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRAModel=Qwen2.5 3B, #Param.=59.87 M, rank=32, Evaluation Protocol=Zero-shot2025.09 | 82.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fine-tuneModel=Qwen2.5 3B, #Param.=3151.91 M, Evaluation Protocol=Zero-shot2025.09 | 82.72 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| StructGPTBackbone=Qwen2.5 32B, Category=Retrieval-based2026.01 | 82.43 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SDFTBackbone=Gemma2 9B, Category=SFT-based2026.01 | 82.42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AugGPTBackbone=Gemma2 9B, Category=Augmented-based2026.01 | 82.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRAModel=Qwen2.5 3B, #Param.=14.97 M, rank=8, Evaluation Protocol=Zero-shot2025.09 | 82.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdapterModel=Qwen2.5 3B, #Param.=67.10 M, Evaluation Protocol=Zero-shot2025.09 | 81.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReGiFTBackbone=Qwen-3-4B2026.01 | 81.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3.1 8B InstClassifier=Self-labeled2026.01 | 81.9 | — | — | -0.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DMTBackbone=Gemma2 9B, Category=SFT-based2026.01 | 81.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3.1 8B InstClassifier=Majority-labeled2026.01 | 81.8 | — | — | -0.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IA3Model=Qwen2.5 3B, #Param.=1.35 M, Evaluation Protocol=Zero-shot2025.09 | 81.76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DiVA-Llama3.1-8BInput Modality=Text, Parameters=8B2025.10 | 81.7 | — | 0.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdaLoRAModel=Qwen2.5 3B, #Param.=61.32 M, rank=32, Evaluation Protocol=Zero-shot2025.09 | 81.66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DMTBackbone=Qwen2.5 32B, Category=SFT-based2026.01 | 81.48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KALEBackbone=Llama3 8B2026.01 | 81.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KALEBackbone=LlaMA3 8B, Category=Augmented-based2026.01 | 81.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoTBackbone=Gemma2 9B, Category=Prompt-based2026.01 | 81.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFTBackbone=Gemma2 9B, Category=SFT-based2026.01 | 81.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VERAModel=Qwen2.5 3B, #Param.=1.42 M, rank=1024, Evaluation Protocol=Zero-shot2025.09 | 81.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral Nemo Base 2407Classifier=Self-labeled2026.01 | 81 | — | — | -0.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral Nemo Base 2407Classifier=Self-labeled2026.01 | 81 | — | — | -0.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SDFTBackbone=Qwen2.5 32B, Category=SFT-based2026.01 | 80.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral Nemo Base 2407Classifier=Majority-labeled2026.01 | 80.7 | — | — | -0.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral Nemo Base 2407Classifier=Majority-labeled2026.01 | 80.7 | — | — | -0.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TOGBackbone=Qwen2.5 32B, Category=Retrieval-based2026.01 | 80.55 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemRewardBackbone=Qwen2.5-3B-Instruct, Supervision Paradigm=20% GT + GNN2026.03 | 80.44 | — | -2.1 | — | — | — | 77.02 | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 1.7B BaseClassifier=Self-labeled2026.01 | 80.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3 1.7B BaseClassifier=Majority-labeled2026.01 | 80.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ours r=1Model=Llama-3.1-8B, Params (%)=0.003%2026.02 | 80.3 | — | — | — | — | -0.3 | — | — | — | — | — | — | — | — | — | — | — | |
| GraphRAGBackbone=Qwen2.5 32B, Category=Retrieval-based2026.01 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MeanLearnBackbone=Gemma2 9B, Category=SFT-based2026.01 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| KG-SFTBackbone=Gemma2 9B, Category=SFT-based2026.01 | 80.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3.1 8BClassifier=Self-labeled2026.01 | 80.1 | — | — | 0.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3.1 8BClassifier=Self-labeled2026.01 | 80.1 | — | — | 0.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ours (1-vec)Model=Llama-3.1-8B, Params (%)=0.001%2026.02 | 80 | — | — | — | — | -0.6 | — | — | — | — | — | — | — | — | — | — | — | |
| R1-pBackbone=Qwen2.5-3B-Instruct, Supervision Paradigm=20% GT2026.03 | 80 | — | -3.45 | — | — | — | 75.67 | — | — | — | — | — | — | — | — | — | — | |
| VanillaBackbone=Gemma2 9B, Category=Prompt-based2026.01 | 79.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3.1 8BClassifier=Majority-labeled2026.01 | 79.7 | — | — | 0.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3.1 8BClassifier=Majority-labeled2026.01 | 79.7 | — | — | 0.3 | — | — | — | — | — | — | — | — | — | — | — | — | — |