Rule Following on IHEval Single-Turn
88.5Accuracy (Reference)Qwen3-4B-it-NS
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Qwen3-4B-it-NSBackbone=Qwen3-4B-it, Inference variant=Neuro-symbolic (NS), Temperature=0.7, top-p=0.8, top-k=202026.04 | 88.5 | 84.9 | 33.8 | 68.6 | |
| Qwen3-4B-itBackbone=Qwen3-4B-it, Temperature=0.7, top-p=0.8, top-k=20, Inference Engine=vLLM2026.04 | 88.4 | 84.3 | 33.5 | 67.7 | |
| Qwen3-4B-it-NSHA-DPOBackbone=Qwen3-4B-it, Training Variant=NSHA-DPO, Temperature=0.7, top-p=0.8, top-k=202026.04 | 88.3 | 87.6 | 47.6 | 72.6 | |
| Qwen3-4B-it-CoTBackbone=Qwen3-4B-it, Prompting=Chain-of-thought (CoT), Temperature=0.7, top-p=0.8, top-k=202026.04 | 88 | 85.9 | 34.9 | 66.6 | |
| Qwen3-4B-it-NSHA-HCALBackbone=Qwen3-4B-it, Training Variant=NSHA-HCAL, Temperature=0.7, top-p=0.8, top-k=202026.04 | 87.8 | 85.1 | 35.5 | 68 | |
| Qwen3-4B-it-NSHA-SFTBackbone=Qwen3-4B-it, Training Variant=NSHA-SFT, Temperature=0.7, top-p=0.8, top-k=202026.04 | 82.5 | 78.2 | 61.6 | 68.6 | |
| Llama3.1-8B-NSBackbone=Llama3.1-8B-Instruct, Inference variant=Neuro-symbolic (NS), Temperature=0.7, top-p=0.8, top-k=202026.04 | 79.8 | 69.5 | 14.6 | 56.4 | |
| Llama3.1-8BBackbone=Llama3.1-8B-Instruct, Temperature=0.7, top-p=0.8, top-k=202026.04 | 79.6 | 71 | 13.8 | 55.8 | |
| Llama3.1-8B-NSHA-HCALBackbone=Llama3.1-8B-Instruct, Training Variant=NSHA-HCAL, Temperature=0.7, top-p=0.8, top-k=202026.04 | 77.4 | 70.3 | 14.7 | 55.3 | |
| Llama3.1-8B-NSHA-DPOBackbone=Llama3.1-8B-Instruct, Training Variant=NSHA-DPO, Temperature=0.7, top-p=0.8, top-k=202026.04 | 76.3 | 74.2 | 41.9 | 64.2 | |
| Llama3.1-8B-CoTBackbone=Llama3.1-8B-Instruct, Prompting=Chain-of-thought (CoT), Temperature=0.7, top-p=0.8, top-k=202026.04 | 75.7 | 65.3 | 18.4 | 54.5 | |
| Llama3.1-8B-NSHA-SFTBackbone=Llama3.1-8B-Instruct, Training Variant=NSHA-SFT, Temperature=0.7, top-p=0.8, top-k=202026.04 | 27 | 23.3 | 22.9 | 27 |