Commonsense Reasoning on ARC-C
96.3AccuracyClaude 3.5 Sonnet
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Claude 3.5 SonnetEvaluation Protocol=Closed-Source API, Emission (gCO2/q)=3.85†, Throughput (Tok/s)=62.12026.03 | 96.3 | — | — | — | |
| GPT-4oEvaluation Protocol=Closed-Source API, Emission (gCO2/q)=4.52†, Throughput (Tok/s)=55.22026.03 | 96.1 | — | — | — | |
| Gemini 2.5 ProEvaluation Protocol=Closed-Source API, Emission (gCO2/q)=3.90†, Throughput (Tok/s)=58.42026.03 | 95.8 | — | — | — | |
| FVAE-LoRAParams (M)=33.532026.06 | 93.18 | 4.19 | 0.27 | — | |
| EcoThinkEvaluation Protocol=Adaptive Inference, Emission (gCO2/q)=1.32, Throughput (Tok/s)=148.62026.03 | 93.1 | — | — | — | |
| BaRAParams (M)=5.6082026.06 | 92.23 | 3.54 | 0.25 | — | |
| DIVERBase Model=Qwen2.5-7B-Base2025.09 | 91.1 | — | — | — | |
| ScalaBLParams (M)=3.7692026.06 | 90.16 | 5.03 | 0.31 | — | |
| GRPOBase Model=Qwen2.5-7B-Base, Reward Strategy=Clip-higher2025.09 | 89.8 | — | — | — | |
| MLEParams (M)=3.7682026.06 | 89.53 | 10.11 | 1.05 | — | |
| BLoBParams (M)=5.4032026.06 | 89.53 | 4.03 | 0.3 | — | |
| C-LoRAParams (M)=4.2752026.06 | 89.52 | 9.05 | 0.62 | — | |
| MC-DropoutParams (M)=3.7682026.06 | 89.44 | 10.06 | 1.02 | — | |
| EnsembleParams (M)=11.3052026.06 | 89.44 | 10.13 | 0.75 | — | |
| GRPOBase Model=LLaMA-3.1-8B-Instruct, Reward Strategy=Clip-higher2025.09 | 89.2 | — | — | — | |
| MAPParams (M)=3.7682026.06 | 88.98 | 10.54 | 1.05 | — | |
| FrugalGPT (Cascade)Evaluation Protocol=Standard CoT, Emission (gCO2/q)=1.95, Throughput (Tok/s)=88.52026.03 | 88.4 | — | — | — | |
| DIVERBase Model=LLaMA-3.1-8B-Instruct2025.09 | 88.2 | — | — | — | |
| CoK + SCBase Model=gpt-3.5-turbo2023.06 | 87.5 | — | — | — | |
| Qwen-3-8B-InstructEvaluation Protocol=Standard CoT, Emission (gCO2/q)=2.12, Throughput (Tok/s)=98.52026.03 | 87.5 | — | — | — | |
| CoK + SC + F2-VBase Model=gpt-3.5-turbo2023.06 | 87.4 | — | — | — | |
| CoK + F2-VBase Model=gpt-3.5-turbo2023.06 | 86.6 | — | — | — | |
| Manual CoT + SCBase Model=gpt-3.5-turbo2023.06 | 86.5 | — | — | — | |
| CoKBase Model=gpt-3.5-turbo2023.06 | 85.7 | — | — | — | |
| DIVERBase Model=DeepSeek-R1-Distill-Qwen-7B2025.09 | 85.6 | — | — | — | |
| LaplaceParams (M)=3.7682026.06 | 85.25 | 37.9 | 0.8 | — | |
| GRPOBase Model=DeepSeek-R1-Distill-Qwen-7B, Reward Strategy=Clip-higher2025.09 | 85.2 | — | — | — | |
| Manual CoTBase Model=gpt-3.5-turbo2023.06 | 84.9 | — | — | — | |
| Llama-3.1-8B-InstructEvaluation Protocol=Standard CoT, Emission (gCO2/q)=2.15, Throughput (Tok/s)=95.82026.03 | 84.1 | — | — | — | |
| PoLAR-VBLL (w/o LA)Backbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 81.79 | 5.86 | 0.64 | — | |
| PoLAR-VBLLBackbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 81.79 | 4.92 | 0.58 | — | |
| PoLAR-MLEBackbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 81.21 | 16.99 | 1.02 | — | |
| PoLAR-LABackbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 81.21 | 9.69 | 0.73 | — | |
| PoLAR-LA-LLBackbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 81.21 | 8.36 | 0.61 | — | |
| MLEBackbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 81.05 | 17.95 | 1.75 | — | |
| LABackbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 81.05 | 9.72 | 1.04 | — | |
| TFB-LLBackbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 80.58 | 10.43 | 0.64 | — | |
| TFBBackbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 80.31 | 4.96 | 0.58 | — | |
| BLoB(N=10)Backbone=Llama3.1-8B, N=102024.10 | 80.081 | 6.811 | — | — | |
| PoLAR-BLoBBackbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 80.03 | 7.02 | 0.59 | — | |
| UQ4CTBackbone=Llama3.1-8B2024.10 | 79.601 | 4.43 | — | — | |
| BLoBBackbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 79.42 | 7.34 | 0.59 | — | |
| BLoB(Mean)Backbone=Llama3.1-8B, Variant=Mean2024.10 | 79.37 | 11.261 | — | — | |
| C-LoRABackbone=Llama-3.1-8B, Fine-tuning steps=50002026.04 | 78.95 | 8.13 | 0.9 | — | |
| FFTModel=Llama-3.2-3B, #Params=3.21B2025.09 | 78.81 | — | — | — | |
| EnsembleBackbone=Llama3.1-8B2024.10 | 78.81 | 13.711 | — | — | |
| BoHAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 78.78 | — | — | — | |
| MixLoRABackbone=Llama3.1-8B2024.10 | 78.561 | 13.711 | — | — | |
| LoRABackbone=Llama3.1-8B2024.10 | 78.21 | 14.07 | — | — | |
| LABackbone=Llama3.1-8B2024.10 | 78.003 | 8.924 | — | — | |
| ABBAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 77.87 | — | — | — | |
| HiRAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 77.61 | — | — | — | |
| MC DropBackbone=Llama3.1-8B2024.10 | 77.14 | 14.12 | — | — | |
| GraLoRAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 76 | — | — | — | |
| Fine-tuning2023.06 | 75 | — | — | — | |
| LoRAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 74.32 | — | — | — | |
| Base ModelBackbone=Llama3.1-8B2024.10 | 74.32 | 11.3 | — | — | |
| rsLoRAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 74.18 | — | — | — | |
| DoRAModel=Llama-3.2-3B, #Params=49.40M2025.09 | 74.15 | — | — | — | |
| PiSSAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 73.52 | — | — | — | |
| CoK + F2-VBase Model=text-davinci-0022023.06 | 73 | — | — | — | |
| CoKBase Model=text-davinci-0022023.06 | 71.1 | — | — | — | |
| BLoBBackbone=Llama2-7B, Method=LoRA, Steps=5,000, N=02024.06 | 70.83 | 20.61 | 1.19 | — | |
| Manual CoTBase Model=text-davinci-0022023.06 | 69.9 | — | — | — | |
| L-LoRA-SParameters=10M, Phase=MAP2026.05 | 69.8 | 27.1 | 1.87 | — | |
| MLEBackbone=Llama2-7B, Method=LoRA, Steps=5,0002024.06 | 69.1 | 29 | 2.85 | — | |
| BLoBBackbone=Llama2-7B, Method=LoRA, Steps=5,000, N=102024.06 | 68.81 | 9.59 | 0.78 | — | |
| L-LoRA-SParameters=10M, Phase=Bayesian2026.05 | 68.8 | 6.4 | 0.88 | — | |
| MCDBackbone=Llama2-7B, Method=LoRA, Steps=5,0002024.06 | 68.69 | 27.53 | 2.67 | — | |
| Few-Shot SPBase Model=text-davinci-0022023.06 | 68.2 | — | — | — | |
| BBBBackbone=Llama2-7B, Method=LoRA, Steps=5,0002024.06 | 68.13 | 26.23 | 2.23 | — | |
| MAPBackbone=Llama2-7B, Method=LoRA, Steps=5,0002024.06 | 67.59 | 29.42 | 2.66 | — | |
| BLoBBackbone=Llama2-7B, Method=LoRA, Steps=5,000, N=52024.06 | 67.34 | 11.22 | 0.9 | — | |
| LAPBackbone=Llama2-7B, Method=LoRA, Steps=5,000, dropout=0.1, learning rate=5e-5, early stopping=50002024.06 | 66.78 | 16.25 | 1.03 | — | |
| Laplace (LA)Phase=MAP2026.05 | 66.3 | 31 | 3.28 | — | |
| ENSBackbone=Llama2-7B, Method=LoRA, Steps=5,0002024.06 | 66.2 | 29.16 | 2.46 | — | |
| EPTparams/task=3.3M, Backbone=Llama2-7B2026.03 | 66.2 | — | — | — | |
| L-LoRA-SParameters=4M, Phase=Bayesian2026.05 | 65.4 | 10 | 0.99 | — | |
| Laplace (LA)Phase=Bayesian2026.05 | 65.3 | 7.4 | 0.88 | — | |
| Zero-Shot CoTBase Model=text-davinci-0022023.06 | 64.7 | — | — | — | |
| MoREparams/task=4.5M, Backbone=Llama2-7B2026.03 | 64.5 | — | — | — | |
| L-LoRA-SParameters=4M, Phase=MAP2026.05 | 64.4 | 21.9 | 1.33 | — | |
| LoRAparams/task=2.1M, Backbone=Llama2-7B2026.03 | 63.5 | — | — | — | |
| L-LoRA-XSRank=100, Phase=MAP2026.05 | 63.1 | 23.9 | 1.45 | — | |
| L-LoRA-XSRank=100, Phase=Bayesian2026.05 | 63.1 | 9.6 | 0.98 | — | |
| BoHAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 62.34 | — | — | — | |
| B-LoRA-XSRank=64, Phase=Bayesian2026.05 | 62.3 | 20.5 | 1.46 | — | |
| ABBAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 62.29 | — | — | — | |
| FFTModel=Llama-3.2-1B, #Params=1.24B2025.09 | 62.05 | — | — | — | |
| MultiLoRAparams/task=10M, Backbone=Llama2-7B2026.03 | 61.9 | — | — | — | |
| B-LoRA-XSRank=64, Phase=MAP2026.05 | 61.8 | 29.9 | 1.97 | — | |
| PiSSAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 61.43 | — | — | — | |
| DoRAModel=Llama-3.2-1B, #Params=22.92M2025.09 | 61.09 | — | — | — | |
| L-LoRA-XSRank=64, Phase=Bayesian2026.05 | 61 | 7.9 | 0.97 | — | |
| GraLoRAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 60.92 | — | — | — | |
| LoRAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 60.49 | — | — | — | |
| rsLoRAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 59.85 | — | — | — | |
| GRPOBase Model=Qwen2.5-Math-1.5B, Reward Strategy=Clip-higher2025.09 | 59.5 | — | — | — | |
| HiRAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 58.7 | — | — | — | |
| L-LoRA-XSRank=64, Phase=MAP2026.05 | 58.3 | 23.3 | 1.3 | — |