Math & Science on MATH 4-shot
69.6ScoreQwen3-8B
Evaluation Results
| Method | Links | |
|---|---|---|
| Qwen3-8BInstruct Model=true, Model Category=AR Baseline, Shot Configuration=4-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=5122025.12 | 69.6 | |
| WeDLM-8BInstruct Model=true, Model Category=WeDLM, Shot Configuration=4-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 64.8 | |
| WeDLM-7BInstruct Model=true, Model Category=WeDLM, Shot Configuration=4-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 55.4 | |
| UltraMix-190kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-190k2025.11 | 49.55 | |
| UltraMix-187kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-187k2025.11 | 48.79 | |
| UltraMix-170kBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=UM-170k2025.11 | 47.55 | |
| Qwen2.5-7BInstruct Model=true, Model Category=AR Baseline, Shot Configuration=4-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=5122025.12 | 45 | |
| SDAR-8BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=4-shot, Inference Framework=JetEngine, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 43.4 | |
| TuluDPOBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=TuluDPO2025.11 | 43.13 | |
| Dream-7BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=4-shot, Inference Framework=dInfer, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 41 | |
| SFTBase Model=Qwen-2.5-7B-TuluSFT, Training Mixture/Protocol=SFT2025.11 | 36.21 | |
| LLaDA-8BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=4-shot, Inference Framework=dInfer, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 34.2 | |
| UltraMixTraining Method=DPO, Dataset=UM-190k, Evaluation Protocol=4-shot2025.11 | 32.5 | |
| UltraMixTraining Method=DPO, Dataset=UM-187k, Evaluation Protocol=4-shot2025.11 | 32.07 | |
| TuluDPOTraining Method=DPO, Dataset=TuluDPO, Evaluation Protocol=4-shot2025.11 | 31.21 | |
| UltraMixTraining Method=DPO, Dataset=UM-170k, Evaluation Protocol=4-shot2025.11 | 30.98 | |
| ORPOTraining Method=ORPO, Dataset=ORPO, Evaluation Protocol=4-shot2025.11 | 29.68 | |
| HelpSteerTraining Method=DPO, Dataset=HelpSteer, Evaluation Protocol=4-shot2025.11 | 27.95 | |
| UltraFBTraining Method=DPO, Dataset=UltraFB, Evaluation Protocol=4-shot2025.11 | 27.19 | |
| CodePrefTraining Method=DPO, Dataset=CodePref, Evaluation Protocol=4-shot2025.11 | 24.46 | |
| SmolLM-3-3B-SFTTraining Method=SFT, Evaluation Protocol=4-shot2025.11 | 24.04 | |
| UltraMix-190kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-190k2025.11 | 23.56 | |
| TuluDPOBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=TuluDPO2025.11 | 22.66 | |
| UltraMix-187kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-187k2025.11 | 22.03 | |
| UltraMix-170kBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=UM-170k2025.11 | 21.22 | |
| Instella-3B-SFT (UM-190k)DPO Training Dataset=UM-190k, Evaluation Protocol=4-shot2025.11 | 16.88 | |
| Instella-3B-SFT (UM-187k)DPO Training Dataset=UM-187k, Evaluation Protocol=4-shot2025.11 | 16.47 | |
| Instella-3B-SFT (TuluDPO)DPO Training Dataset=TuluDPO, Evaluation Protocol=4-shot2025.11 | 15.71 | |
| Instella-3B-SFT (UM-170k)DPO Training Dataset=UM-170k, Evaluation Protocol=4-shot2025.11 | 15.56 | |
| Instella-3B-SFT (ORPO)DPO Training Dataset=ORPO, Evaluation Protocol=4-shot2025.11 | 14.39 | |
| Instella-3B-SFT (UltraFB)DPO Training Dataset=UltraFB, Evaluation Protocol=4-shot2025.11 | 13.52 | |
| Instella-3B-SFT (HelpSteer)DPO Training Dataset=HelpSteer, Evaluation Protocol=4-shot2025.11 | 12.76 | |
| Instella-3B-SFT (SFT)DPO Training Dataset=None (SFT Baseline), Evaluation Protocol=4-shot2025.11 | 12.31 | |
| Instella-3B-SFT (CodePref)DPO Training Dataset=CodePref, Evaluation Protocol=4-shot2025.11 | 12.08 | |
| SFTBase Model=Llama-3.1-8B-TuluSFT, Training Mixture/Protocol=SFT2025.11 | 12.08 |