Multistep Soft Reasoning on MuSR
69AccuracyGPT-OSS-120B
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| GPT-OSS-120BOrganization=Flat2026.04 | 69 | 12,700 | — | — | |
| OrgAgentBackbone=GPT-5mini, Organization=Hierarchical (AUTO)2026.04 | 64.83 | 7,195 | 3.81 | 46.38 | |
| GPT-5miniOrganization=Flat2026.04 | 62.45 | 13,419 | — | — | |
| OrgAgentBackbone=GPT-OSS-120B, Organization=Hierarchical (AUTO)2026.04 | 59.5 | 5,994 | -13.77 | 52.8 | |
| GPT-OSS-120BOrganization=Baseline2026.04 | 50.65 | 1,328 | — | — | |
| SmoothQuantModel=Qwen2.5-7B, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 46.39 | — | — | — | |
| QuaRotModel=Qwen2.5-7B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 45.58 | — | — | — | |
| MixLLMModel=Qwen2.5-7B, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 44.91 | — | — | — | |
| MixLLMModel=Qwen2.5-7B, Precision=W4.4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 44.79 | — | — | — | |
| float16Model=Qwen2.5-7B, Precision=float16, Evaluation Protocol (Shots)=0-shot2024.12 | 44.51 | — | — | — | |
| MixLLMModel=Mistral-7B-v0.3, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 43.19 | — | — | — | |
| MixLLMModel=Qwen2.5-7B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 43.11 | — | — | — | |
| SmoothQuantModel=Average, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 42.06 | — | — | — | |
| MixLLMModel=Average, Precision=W4.4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 41.74 | — | — | — | |
| MixLLMModel=Average, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 41.7 | — | — | — | |
| QuaRotModel=Average, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 41.65 | — | — | — | |
| QServeModel=Qwen2.5-7B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 41.59 | — | — | — | |
| MixLLMModel=Mistral-7B-v0.3, Precision=W4.4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 41.11 | — | — | — | |
| float16Model=Average, Precision=float16, Evaluation Protocol (Shots)=0-shot2024.12 | 41.07 | — | — | — | |
| MixLLMModel=Average, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 40.94 | — | — | — | |
| SmoothQuantModel=Mistral-7B-v0.3, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 40.73 | — | — | — | |
| float16Model=Mistral-7B-v0.3, Precision=float16, Evaluation Protocol (Shots)=0-shot2024.12 | 40.72 | — | — | — | |
| QuaRotModel=Qwen2.5-7B, Precision=W4A4, Evaluation Protocol (Shots)=0-shot2024.12 | 40.68 | — | — | — | |
| MixLLMModel=Mistral-7B-v0.3, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 40.59 | — | — | — | |
| QServeModel=Mistral-7B-v0.3, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 40.57 | — | — | — | |
| QuaRotModel=Mistral-7B-v0.3, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 40.32 | — | — | — | |
| QServeModel=Average, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 39.92 | — | — | — | |
| QuaRotModel=Mistral-7B-v0.3, Precision=W4A4, Evaluation Protocol (Shots)=0-shot2024.12 | 39.77 | — | — | — | |
| QuaRotModel=Average, Precision=W4A4, Evaluation Protocol (Shots)=0-shot2024.12 | 39.46 | — | — | — | |
| MixLLMModel=Llama-3.1-8B, Precision=W4.4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 39.32 | — | — | — | |
| SmoothQuantModel=Llama-3.1-8B, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 39.05 | — | — | — | |
| QuaRotModel=Llama-3.1-8B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 39.05 | — | — | — | |
| MixLLMModel=Llama-3.1-8B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 38.81 | — | — | — | |
| float16Model=Llama-3.1-8B, Precision=float16, Evaluation Protocol (Shots)=0-shot2024.12 | 37.99 | — | — | — | |
| QuaRotModel=Llama-3.1-8B, Precision=W4A4, Evaluation Protocol (Shots)=0-shot2024.12 | 37.92 | — | — | — | |
| QServeModel=Llama-3.1-8B, Precision=W4A8, Evaluation Protocol (Shots)=0-shot2024.12 | 37.6 | — | — | — | |
| LLaMA-3.1-8BOrganization=Flat2026.04 | 37.41 | 25,600 | — | — | |
| MixLLMModel=Llama-3.1-8B, Precision=W8A8, Evaluation Protocol (Shots)=0-shot2024.12 | 37.32 | — | — | — | |
| OrgAgentBackbone=LLaMA-3.1-8B, Organization=Hierarchical (AUTO)2026.04 | 34 | 5,912 | -9.12 | 76.91 | |
| GPT-5miniOrganization=Baseline2026.04 | 29 | 1,603 | — | — | |
| LLaMA-3.1-8BOrganization=Baseline2026.04 | 10.33 | 1,061 | — | — |