Structured JSON Generation on JSON Generation dataset
99.8Structural AccuracyQwen3-4B + Outlines
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Qwen3-4B + OutlinesProtocol=SFT, Decoding=FSM-based constrained decoding2025.11 | 99.8 | 100 | 99.5 | 99.8 | 79.5 | 15.2 | 2.1 | |
| RL-StructProtocol=RL-Struct (proposed)2025.11 | 89.7 | 92.1 | 85.3 | 89.7 | 84.5 | 90.6 | 1.5 | |
| Qwen3-4B + PPOProtocol=PPO2025.11 | 89.1 | 91.5 | 84.8 | 89 | 84.2 | 90.4 | 1.8 | |
| Qwen3-4B + DPOProtocol=DPO2025.11 | 82.5 | 88.4 | 83 | 81.5 | 82 | 90.2 | 6.5 | |
| LLaMA-3-8BProtocol=SFT2025.11 | 78.2 | 85.4 | 81.2 | 78.2 | 86 | 40.1 | 9.3 | |
| Phi-3-miniProtocol=SFT2025.11 | 74.1 | 84.8 | 79.5 | 74.1 | 81.5 | 85.4 | 8.1 | |
| Qwen3-4BProtocol=SFT2025.11 | 65.4 | 72.1 | 68.9 | 68.5 | 80 | 90.8 | 14.1 | |
| Mistral-7BProtocol=Zero-shot2025.11 | 52.3 | 83.5 | 76.2 | 60.5 | 85 | 45.2 | 12.5 | |
| GPT-3.5Protocol=Zero-shot2025.11 | 45.5 | 82.1 | 75 | 55.2 | 88 | — | 15.2 |