Language Reasoning on BBH (BIG-Bench Hard)
87.5Average BBH ScorePREFPO
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PREFPOLabeled=true2026.03 | 87.5 | 97.8 | 85.9 | 76.1 | 91.5 | 81.8 | 91.7 | 96.9 | 91.9 | 73.5 | — | — | — | — | — | — | — | |
| MIPROv2Setting=auto=heavy, Base Model=GPT-52026.03 | 87.3 | 99.4 | 95.6 | 75.1 | 89.9 | 77.6 | 90.2 | 99.9 | 86.2 | 71.4 | — | — | — | — | — | — | — | |
| GEPASetting=auto=heavy, Reflection Model=GPT-52026.03 | 87 | 98.7 | 94.8 | 77.8 | 90.3 | 69.8 | 92.1 | 97.3 | 91.2 | 71.4 | — | — | — | — | — | — | — | |
| TextGradReflection Model=GPT-5, Iterations=152026.03 | 85 | 98.7 | 91.5 | 73.2 | 76.5 | 78 | 91.1 | 94.6 | 89.6 | 71.5 | — | — | — | — | — | — | — | |
| PREFPOLabeled=false2026.03 | 83.9 | 97.1 | 77.4 | 76.4 | 80.7 | 67.7 | 91.5 | 96.6 | 92.9 | 74.9 | — | — | — | — | — | — | — | |
| GPT-4oPrompting Strategy=few-shot + CoT2026.03 | 82.9 | 99.2 | 89.6 | 72.7 | 67.6 | 85.6 | 80 | 96 | 86.8 | 68.8 | — | — | — | — | — | — | — | |
| GPT-5Prompting Strategy=zero shot2026.03 | 75 | 9.2 | 82.4 | 70.6 | 82 | 72 | 100 | 78.4 | 98 | 82 | — | — | — | — | — | — | — | |
| GPT-4oPrompting Strategy=zero shot2026.03 | 71.3 | 74.8 | 73.2 | 72.2 | 64.4 | 54.8 | 68.8 | 77.2 | 87.2 | 68.8 | — | — | — | — | — | — | — | |
| LMNetBackbone=Qwen2.5-0.5B2025.05 | 47.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Self-RefineBackbone=Qwen2.5-0.5B, Passes=162025.05 | 24.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRAHubBackbone=LLaMA-3.1-8B, Stage=Training, Hardware=NVIDIA H1002025.11 | 24.28 | 23.63 | — | 47.59 | — | — | — | 16.2 | 25.29 | — | — | 24.28 | 24.52 | 21.87 | 17.61 | 17.51 | — | |
| Self-ConsistencyBackbone=Qwen2.5-0.5B, Samples=162025.05 | 21.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PromptBackbone=Qwen2.5-0.5B2025.05 | 20.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFTBackbone=Qwen2.5-0.5B2025.05 | 19.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LOGO (Norm)Backbone=LLaMA-3.1-8B, Stage=Inference, Merging Signal=Norm, Hardware=NVIDIA H1002025.11 | 2.08 | 2.36 | — | 1.98 | — | — | — | 1.86 | 1.77 | — | 2.08 | — | 2.87 | 1.86 | 1.62 | 2.32 | — | |
| LoRARetrieverBackbone=LLaMA-3.1-8B, Stage=Inference, Hardware=NVIDIA H1002025.11 | 2.03 | 2.06 | — | 1.91 | — | — | — | 1.99 | 1.85 | — | 2.03 | — | 1.83 | 1.89 | 2.38 | 2.29 | — | |
| LOGO (Entropy)Backbone=LLaMA-3.1-8B, Stage=Inference, Merging Signal=Entropy, Hardware=NVIDIA H1002025.11 | 1.87 | 1.75 | — | 2.07 | — | — | — | 1.67 | 1.65 | — | 1.87 | — | 1.73 | 1.97 | 1.93 | 2.22 | — | |
| AdapterSoupBackbone=LLaMA-3.1-8B, Stage=Inference, Hardware=NVIDIA H1002025.11 | 1.8 | 4.78 | — | 0.99 | — | — | — | 1.24 | 0.67 | — | 1.8 | — | 2.27 | 1.71 | 1.91 | 0.83 | — | |
| LoRAHubBackbone=LLaMA-3.1-8B, Stage=Inference, Hardware=NVIDIA H1002025.11 | 1.15 | 1.57 | — | 0.36 | — | — | — | 0.73 | 0.39 | — | 1.15 | — | 1.76 | 0.92 | 1.85 | 1.61 | — | |
| BaseBackbone=LLaMA-3.1-8B, Stage=Inference, Hardware=NVIDIA H1002025.11 | 0.47 | 0.41 | — | 0.32 | — | — | — | 0.72 | 0.44 | — | 0.47 | — | 0.37 | 0.4 | 0.38 | 0.76 | — | |
| DeepSeek-V3.2 Base# Shots=3-shot, Architecture=MoE, # Activated Params=37B, # Total Params=671B2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 87.6 | |
| DeepSeek-V4-Flash Base# Shots=3-shot, Architecture=MoE, # Activated Params=13B, # Total Params=284B2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 86.9 | |
| DeepSeek-V4-Pro Base# Shots=3-shot, Architecture=MoE, # Activated Params=49B, # Total Params=1.6T2026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 87.5 |