Bias Evaluation on BBQ
99.3AccuracyC2PO
Evaluation results
| Method | Links | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| C2POBackbone=DeepSeek2025.12 | 99.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| C2POBackbone=DeepSeek-R1-Distill-Qwen-14B2025.12 | 99.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.7 | — | — | — | — | — | — | — | |
| GRPOBackbone=DeepSeek-R1-Distill-Qwen-14B2025.12 | 97.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.4 | — | — | — | — | — | — | — | |
| C2POBackbone=LLaMA-2-13B-Chat2025.12 | 97.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.6 | — | — | — | — | — | — | — | |
| Φ-SEABackbone=Mistral-7B2024.05 | 95.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.5 | 3.9 | — | — | — | — | — | — | — | — | — | |
| CPOBackbone=DeepSeek-R1-Distill-Qwen-14B2025.12 | 95.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 7.6 | — | — | — | — | — | — | — | |
| Linear-SEABackbone=Mistral-7B2024.05 | 94.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.7 | 4.3 | — | — | — | — | — | — | — | — | — | |
| ICLBackbone=Mistral-7B2024.05 | 94.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.8 | 4.2 | — | — | — | — | — | — | — | — | — | |
| IndividualBackbone=Qwen-14B2025.12 | 94.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DPOBackbone=DeepSeek-R1-Distill-Qwen-14B2025.12 | 91.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 8.8 | — | — | — | — | — | — | — | |
| GPT-42025.12 | 90.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IPOBackbone=DeepSeek-R1-Distill-Qwen-14B2025.12 | 89.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 8.7 | — | — | — | — | — | — | — | |
| FRBackbone=DeepSeek-R1-Distill-Qwen-14B2025.12 | 88.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 7.9 | — | — | — | — | — | — | — | |
| BCOBackbone=LLaMA-2-13B-Chat2025.12 | 88.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 6.8 | — | — | — | — | — | — | — | |
| BCOBackbone=DeepSeek-R1-Distill-Qwen-14B2025.12 | 88.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 6.8 | — | — | — | — | — | — | — | |
| MARIModel=Qwen2.5-32B2026.05 | 87.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DTS-D*Backbone=Qwen-14B, Variant=Optimized2025.12 | 87.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DTS-T*Backbone=Qwen-14B, Variant=Optimized2025.12 | 87.28 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DTS-TBackbone=Qwen-14B2025.12 | 87.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DTS-DBackbone=Qwen-14B2025.12 | 87.09 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CPOBackbone=LLaMA-2-13B-Chat2025.12 | 86.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 5.8 | — | — | — | — | — | — | — | |
| Twin-MergingBackbone=Qwen-14B2025.12 | 86.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| STEER2ADAPTBackbone=Qwen-2.5-7B-Instruct2026.02 | 86.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EMR-MERGINGBackbone=Qwen-14B2025.12 | 86.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Few-ShotBackbone=Qwen-2.5-7B-Instruct, n=12026.02 | 85.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DAREBackbone=Qwen-14B2025.12 | 85.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Few-ShotBackbone=Qwen-2.5-7B-Instruct, n=22026.02 | 85.62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ties-MergingBackbone=Qwen-14B2025.12 | 85.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Task-ArithmeticBackbone=Qwen-14B2025.12 | 84.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| REPBackbone=Qwen-2.5-7B-Instruct2026.02 | 84.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ICLBackbone=Qwen-2.5-7B-Instruct2026.02 | 84.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Direct InferenceBackbone=Qwen-2.5-7B-Instruct2026.02 | 84.36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAABackbone=Qwen-2.5-7B-Instruct2026.02 | 83.33 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IPOBackbone=LLaMA-2-13B-Chat2025.12 | 82.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 7.2 | — | — | — | — | — | — | — | |
| MARIModel=Qwen2.5-14B2026.05 | 82.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReFTModel=Qwen2.5-32B2026.05 | 82.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T-SwitchBackbone=Qwen-14B2025.12 | 80.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MARIModel=Llama-3-8B2026.05 | 79.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MARIModel=Llama-2-13B2026.05 | 78.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MARIModel=Qwen2-7B2026.05 | 78.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NL-ITIModel=Qwen2.5-32B2026.05 | 77.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ITIModel=Qwen2.5-32B2026.05 | 75.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MARIModel=Llama-2-7B2026.05 | 75.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAAModel=Qwen2.5-32B2026.05 | 74.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VanillaModel=Qwen2.5-32B2026.05 | 73.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ITIModel=Llama-2-13B2026.05 | 71.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GIFTModel Size=32B2025.10 | 71.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| STEER2ADAPTBackbone=Llama-3.1-8B-Instruct2026.02 | 70.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ICLBackbone=Llama-3.1-8B-Instruct2026.02 | 70.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Direct InferenceBackbone=Llama-3.1-8B-Instruct2026.02 | 69.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NL-ITIModel=Qwen2-7B2026.05 | 68.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FRBackbone=LLaMA-2-13B-Chat2025.12 | 67.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 9.3 | — | — | — | — | — | — | — | |
| ReFTModel=Llama-2-13B2026.05 | 67.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReFTModel=Qwen2-7B2026.05 | 67.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Few-ShotBackbone=Llama-3.1-8B-Instruct, n=12026.02 | 67.63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NL-ITIModel=Qwen2.5-14B2026.05 | 67.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| REPBackbone=Llama-3.1-8B-Instruct2026.02 | 67.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TinyAya GlobalScenario=Cultural Adaptation of a Baseline SFT Model2026.06 | 67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ITIModel=Qwen2-7B2026.05 | 66.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAAModel=Qwen2-7B2026.05 | 66.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ITIModel=Qwen2.5-14B2026.05 | 65.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GRPOModel Size=32B2025.10 | 65.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAAModel=Qwen2.5-14B2026.05 | 65.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VanillaModel=Qwen2.5-14B2026.05 | 64.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAABackbone=Mistral-7b-v0.3, Method=CAA2025.08 | 64.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReFTModel=Qwen2.5-14B2026.05 | 64.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MSRSBackbone=Llama3-8b-inst., Method=MSRS2025.08 | 64.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MSRSBackbone=Mistral-7b-v0.3, Method=MSRS2025.08 | 64.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MSRSBackbone=Qwen2-7b-inst., Method=MSRS2025.08 | 64.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CS-ReFTBackbone=Llama3-8b-inst., Method=CS-ReFT2025.08 | 64.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAT-STEERBackbone=Qwen2-7b-inst., Method=MAT-STEER2025.08 | 64.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CS-ReFTBackbone=Qwen2-7b-inst., Method=CS-ReFT2025.08 | 63.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CS-ReFTBackbone=Mistral-7b-v0.3, Method=CS-ReFT2025.08 | 63.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VanillaModel=Qwen2-7B2026.05 | 63.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReFTBackbone=Llama3-8b-inst., Method=ReFT2025.08 | 63.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReFTModel=Llama-3-8B2026.05 | 63.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReFTBackbone=Qwen2-7b-inst., Method=ReFT2025.08 | 63.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAABackbone=Qwen2-7b-inst., Method=CAA2025.08 | 63.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAAModel=Llama-3-8B2026.05 | 63.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2-7b-inst.Backbone=Qwen2-7b-inst., Method=Base2025.08 | 63.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ICLBackbone=Qwen2-7b-inst., Method=ICL2025.08 | 63.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Few-ShotBackbone=Llama-3.1-8B-Instruct, n=22026.02 | 63.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAT-STEERBackbone=Mistral-7b-v0.3, Method=MAT-STEER2025.08 | 63.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NL-ITIModel=Llama-2-13B2026.05 | 63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAABackbone=Llama3-8b-inst., Method=CAA2025.08 | 62.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAAModel=Llama-2-13B2026.05 | 62.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAT-STEERBackbone=Qwen2.5-7B2025.02 | 62.59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NL-ITIModel=Llama-3-8B2026.05 | 62.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAT-STEERBackbone=Llama3-8b-inst., Method=MAT-STEER2025.08 | 62.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ICLBackbone=Mistral-7b-v0.3, Method=ICL2025.08 | 62.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ICLBackbone=Llama3-8b-inst., Method=ICL2025.08 | 61.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAABackbone=Llama-3.1-8B-Instruct2026.02 | 61.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mistral-7b-v0.3Backbone=Mistral-7b-v0.3, Method=Base2025.08 | 61.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReFTBackbone=Mistral-7b-v0.3, Method=ReFT2025.08 | 61.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAT-STEERBackbone=Llama-3.1-8B-Chat2025.02 | 61.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ITIBackbone=Llama3-8b-inst., Method=ITI2025.08 | 61.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ITIModel=Llama-3-8B2026.05 | 61.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama3-8b-inst.Backbone=Llama3-8b-inst., Method=Base2025.08 | 60.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VanillaModel=Llama-3-8B2026.05 | 60.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFTBackbone=Qwen2.5-7B2025.02 | 60.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |