Rule Following on IHEval Multi-Turn
89.8Accuracy (Reference)Qwen3-4B-it
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Qwen3-4B-itBackbone=Qwen3-4B-it, Temperature=0.7, top-p=0.8, top-k=20, Inference Engine=vLLM2026.04 | 89.8 | 83.4 | 26.6 | 67.7 | |
| Qwen3-4B-it-NSHA-HCALBackbone=Qwen3-4B-it, Training Variant=NSHA-HCAL, Temperature=0.7, top-p=0.8, top-k=202026.04 | 89.5 | 83.4 | 26.7 | 68 | |
| Qwen3-4B-it-NSBackbone=Qwen3-4B-it, Inference variant=Neuro-symbolic (NS), Semantic Identification=GPT-5-mini, Temperature=0.7, top-p=0.8, top-k=202026.04 | 89.1 | 82.8 | 32.6 | 68.6 | |
| Qwen3-4B-it-NSHA-DPOBackbone=Qwen3-4B-it, Training Variant=NSHA-DPO, Temperature=0.7, top-p=0.8, top-k=202026.04 | 89 | 86.6 | 36.6 | 72.6 | |
| Qwen3-4B-it-CoTBackbone=Qwen3-4B-it, Prompting=Chain-of-thought (CoT), Temperature=0.7, top-p=0.8, top-k=20, Inference Engine=vLLM2026.04 | 88.8 | 76.7 | 25.2 | 66.6 | |
| Llama3.1-8BBackbone=Llama3.1-8B-Instruct, Temperature=0.7, top-p=0.8, top-k=20, Inference Engine=vLLM2026.04 | 81.6 | 68.8 | 20.3 | 55.8 | |
| Qwen3-4B-it-NSHA-SFTBackbone=Qwen3-4B-it, Training Variant=NSHA-SFT, Temperature=0.7, top-p=0.8, top-k=202026.04 | 80.3 | 82 | 27.1 | 68.6 | |
| Llama3.1-8B-NSBackbone=Llama3.1-8B-Instruct, Inference variant=Neuro-symbolic (NS), Temperature=0.7, top-p=0.8, top-k=202026.04 | 79.9 | 72.1 | 22.4 | 56.4 | |
| Llama3.1-8B-NSHA-HCALBackbone=Llama3.1-8B-Instruct, Training Variant=NSHA-HCAL, Temperature=0.7, top-p=0.8, top-k=202026.04 | 78.8 | 69 | 21.2 | 55.3 | |
| Llama3.1-8B-NSHA-DPOBackbone=Llama3.1-8B-Instruct, Training Variant=NSHA-DPO, Temperature=0.7, top-p=0.8, top-k=202026.04 | 77.2 | 76.8 | 38.7 | 64.2 | |
| Llama3.1-8B-CoTBackbone=Llama3.1-8B-Instruct, Prompting=Chain-of-thought (CoT), Temperature=0.7, top-p=0.8, top-k=202026.04 | 74.7 | 67.6 | 25.2 | 54.5 | |
| Llama3.1-8B-NSHA-SFTBackbone=Llama3.1-8B-Instruct, Training Variant=NSHA-SFT, Temperature=0.7, top-p=0.8, top-k=202026.04 | 27.4 | 45 | 16.1 | 27 |