Reasoning on MuSR
71.89AccuracyLLaDA2.1-mini
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LLaDA2.1-minimode=Q Mode2026.02 | 71.89 | 1.56 | |
| LLaDA2.0-mini2026.02 | 71.48 | 1.45 | |
| Ling-mini-2.02026.02 | 71.36 | — | |
| LLaDA2.1-minimode=S Mode2026.02 | 70.43 | 2.48 | |
| Qwen3-8Bno think=true2026.02 | 70.02 | — | |
| Baseline with CoTBackbone=Llama 3 70B2026.07 | 66.8 | — | |
| MALLM (Judge)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Judge2026.07 | 64.2 | — | |
| BaselineBackbone=Llama 3 70B2026.07 | 63.2 | — | |
| MALLM (Majority Consensus)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Majority Consensus2026.07 | 61.3 | — | |
| MALLM (Unanimity Consensus)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Unanimity Consensus2026.07 | 61.3 | — | |
| MALLM (Average Consensus)Backbone=Llama 3 70B, Discussion Paradigm=Memory2026.07 | 60.9 | — | |
| MALLM (Supermajority Consensus)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Supermajority Consensus2026.07 | 60.2 | — | |
| MALLM (Cumulative Voting)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Cumulative Voting2026.07 | 60 | — | |
| MALLM (Average Voting)Backbone=Llama 3 70B, Discussion Paradigm=Memory2026.07 | 59.6 | — | |
| MALLM (Ranked Voting)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Ranked Voting2026.07 | 59.5 | — | |
| MALLM (Simple Voting)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Simple Voting2026.07 | 59.3 | — | |
| Qwen3-4BLoss function=base+aux2026.02 | 55.46 | — | |
| Qwen3-4BLoss function=base2026.02 | 54.14 | — | |
| Qwen3-8BLoss function=base+aux2026.02 | 53.5 | — | |
| Qwen3-8BLoss function=base2026.02 | 49.78 | — | |
| MALLM (Approval Voting)Backbone=Llama 3 70B, Discussion Paradigm=Memory, Decision Protocol=Approval Voting2026.07 | 49.1 | — | |
| LLaMA3.1-8BLoss function=base+aux2026.02 | 46.17 | — | |
| LLaMA3.1-8BLoss function=base2026.02 | 44.87 | — | |
| UM-190kBackbone=Apertus-8B-SFT, Evaluation Protocol=0-shot2025.11 | 36.28 | — | |
| UM-187kBackbone=Apertus-8B-SFT, Evaluation Protocol=0-shot2025.11 | 36.07 | — | |
| UltraFBBackbone=Apertus-8B-SFT, Evaluation Protocol=0-shot2025.11 | 35.94 | — | |
| UM-170kBackbone=Apertus-8B-SFT, Evaluation Protocol=0-shot2025.11 | 35.94 | — | |
| SFTBackbone=Apertus-8B-SFT, Evaluation Protocol=0-shot2025.11 | 35.85 | — | |
| TuluDPOBackbone=Apertus-8B-SFT, Evaluation Protocol=0-shot2025.11 | 35.07 | — | |
| ORPOBackbone=Apertus-8B-SFT, Evaluation Protocol=0-shot2025.11 | 34.98 | — | |
| HelpSteerBackbone=Apertus-8B-SFT, Evaluation Protocol=0-shot2025.11 | 34.47 | — | |
| CodePrefBackbone=Apertus-8B-SFT, Evaluation Protocol=0-shot2025.11 | 32.48 | — |