Commonsense Reasoning on COPA (test)
98.67AccuracyOPRO
Evaluation results
| Method | Links | |
|---|---|---|
| OPROModel=Qwen2.5-32B2025.04 | 98.67 | |
| CAPOModel=Qwen2.5-32B2025.04 | 98.47 | |
| CAPOModel=Llama-3.3-70B2025.04 | 98.27 | |
| InitialModel=Qwen2.5-32B2025.04 | 98.27 | |
| PromptWizardModel=Qwen2.5-32B2025.04 | 98.13 | |
| OPROModel=Llama-3.3-70B2025.04 | 98.07 | |
| EvoPromptGAModel=Qwen2.5-32B2025.04 | 97.87 | |
| InitialModel=Llama-3.3-70B2025.04 | 97.65 | |
| EvoPromptGAModel=Llama-3.3-70B2025.04 | 97.6 | |
| OPROModel=Mistral-Small-24B2025.04 | 96.33 | |
| EvoPromptGAModel=Mistral-Small-24B2025.04 | 96.13 | |
| CAPOModel=Mistral-Small-24B2025.04 | 95.13 | |
| InitialModel=Mistral-Small-24B2025.04 | 94.56 | |
| Coherence Boosting (GPT-3 175B)alpha=-0.522021.10 | 94 | |
| GPT-3 175Balpha=02021.10 | 93 | |
| AUSteer-FFNModel=LLaMA2-13B-Chat2026.02 | 91.2 | |
| AUSteer-HeadModel=LLaMA2-13B-Chat2026.02 | 91 | |
| AUSteer-HeadModel=Gemma2-27B-it2026.02 | 90.2 | |
| FedTTFederated Setting=large-scale FL, Backbone=LLaMA2-7B, Number of Parameters=0.52M2024.10 | 90 | |
| AUSteer-FFNModel=Gemma2-27B-it2026.02 | 89.8 | |
| LoRAFederated Setting=cross-silo FL, Backbone=LLaMA2-13B, Number of Parameters=6.55M, Rank=82024.10 | 89 | |
| FedTTFederated Setting=cross-silo FL, Backbone=LLaMA2-13B, Number of Parameters=0.64M2024.10 | 89 | |
| VanillaModel=LLaMA2-13B-Chat2026.02 | 89 | |
| FedTT+Federated Setting=cross-silo FL, Backbone=LLaMA2-13B, Number of Parameters=0.18M2024.10 | 88 | |
| GPT-3 175Balpha=-12021.10 | 87 | |
| LoRAFederated Setting=large-scale FL, Backbone=LLaMA2-7B, Number of Parameters=4.19M, Rank=82024.10 | 87 | |
| AUSteer-HeadModel=LLaMA3.1-8B-Instruct2026.02 | 86 | |
| AUSteer-FFNModel=LLaMA3.1-8B-Instruct2026.02 | 86 | |
| VanillaModel=Gemma2-27B-it2026.02 | 86 | |
| MeZOModel=OPT-13B, Forward pass budget=10,0002026.05 | 86 | |
| ZoVHModel=OPT-13B, N=1, Forward pass budget=10,0002026.05 | 85.67 | |
| VanillaModel=LLaMA3.1-8B-Instruct2026.02 | 83.8 | |
| ZoVHModel=OPT-13B, N=2, Forward pass budget=10,0002026.05 | 83.33 | |
| HiZOOModel=OPT-13B, Forward pass budget=10,0002026.05 | 80.67 | |
| KELMBackbone=BERT-large, Knowledge Source=WordNet2021.09 | 78 | |
| Coherence Boosting (GPT-2 XL)alpha=-0.442021.10 | 77 | |
| ZoVHModel=OPT-1.3B, N=1, Forward pass budget=10,0002026.05 | 76.67 | |
| HiZOOModel=OPT-1.3B, Forward pass budget=10,0002026.05 | 76 | |
| MeZOModel=OPT-1.3B, Forward pass budget=10,0002026.05 | 75.33 | |
| ZoVHModel=OPT-1.3B, N=2, Forward pass budget=10,0002026.05 | 73.67 | |
| GPT-2 XL (1.6B)alpha=02021.10 | 73 | |
| Sasaki et al.2018.10 | 71.2 | |
| BERT-largeBackbone=BERT-large2021.09 | 70.6 | |
| Luo et al.2018.10 | 70.2 | |
| GPT-2 XL (1.6B)alpha=-12021.10 | 70 | |
| AGQsynonyms=None2018.10 | 66.2 | |
| Gordon et al.2018.10 | 65.4 | |
| AGQSsynonyms=WordNet2018.10 | 65.1 | |
| Coherence Boosting (GPT-2 Small)alpha=-0.692021.10 | 64 | |
| Goodwin et al.2018.10 | 63.4 | |
| GPT-2 Small (125M)alpha=02021.10 | 62 | |
| PromptWizardModel=Mistral-Small-24B2025.04 | 57.47 | |
| GPT-2 Small (125M)alpha=-12021.10 | 56 | |
| PromptWizardModel=Llama-3.3-70B2025.04 | 50.33 |