JSON Generation on JSON-mode-eval e=1.1
100Syntax AccuracyTruncProof
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| TruncProofBackbone=Gemma2-2B, Decoding Strategy=Greedy, Prompt Engineering=False2026.05 | 100 | 62 | 21 | 25.7 | 3.9 | |
| TruncProofBackbone=Gemma2-2B, Decoding Strategy=BS, Prompt Engineering=False2026.05 | 100 | 85 | 37 | 60.8 | 39 | |
| TruncProofBackbone=Gemma2-2B, Decoding Strategy=MCTS, Prompt Engineering=False2026.05 | 100 | 86 | 58 | 518.1 | 496.3 | |
| TruncProofBackbone=Llama2-7B-Chat-HF, Decoding Strategy=Greedy, Prompt Engineering=False2026.05 | 100 | 51 | 2 | 19 | 1.4 | |
| TruncProofBackbone=Llama2-7B-Chat-HF, Decoding Strategy=BS, Prompt Engineering=False2026.05 | 100 | 67 | 29 | 37 | 19.4 | |
| TruncProofBackbone=Llama2-7B-Chat-HF, Decoding Strategy=MCTS, Prompt Engineering=False2026.05 | 100 | 70 | 41 | 209.2 | 191.6 | |
| OutlinesBackbone=Gemma2-2B, Decoding Strategy=Greedy, Prompt Engineering=False2026.05 | 36 | 33 | 22 | 458.7 | 436.9 | |
| OutlinesBackbone=Llama2-7B-Chat-HF, Decoding Strategy=Greedy, Prompt Engineering=True2026.05 | 19 | 17 | 5 | — | — | |
| OutlinesBackbone=Llama2-7B-Chat-HF, Decoding Strategy=Greedy, Prompt Engineering=False2026.05 | 18 | 13 | 4 | 72.2 | 54.6 | |
| OutlinesBackbone=Gemma2-2B, Decoding Strategy=Greedy, Prompt Engineering=True2026.05 | 17 | 17 | 8 | — | — | |
| SynCodeBackbone=Llama2-7B-Chat-HF, Decoding Strategy=Greedy, Prompt Engineering=True2026.05 | 16 | 14 | 5 | — | — | |
| XGrammarBackbone=Llama2-7B-Chat-HF, Decoding Strategy=Greedy, Prompt Engineering=True2026.05 | 16 | 13 | 3 | — | — | |
| SynCodeBackbone=Llama2-7B-Chat-HF, Decoding Strategy=Greedy, Prompt Engineering=False2026.05 | 11 | 10 | 4 | 18.4 | 0.8 | |
| XGrammarBackbone=Llama2-7B-Chat-HF, Decoding Strategy=Greedy, Prompt Engineering=False2026.05 | 11 | 9 | 2 | 18.3 | 0.7 | |
| OutlinesBackbone=Llama2-7B-Chat-HF, Decoding Strategy=BS, Prompt Engineering=False2026.05 | 10 | 8 | 4 | 598.8 | 581.2 | |
| XGrammarBackbone=Llama2-7B-Chat-HF, Decoding Strategy=MCTS, Prompt Engineering=False2026.05 | 9 | 8 | 3 | 175.1 | 157.5 | |
| No constraintBackbone=Gemma2-2B, Decoding Strategy=Greedy, Prompt Engineering=True2026.05 | 8 | 8 | 4 | — | — | |
| XGrammarBackbone=Gemma2-2B, Decoding Strategy=Greedy, Prompt Engineering=True2026.05 | 8 | 8 | 4 | — | — | |
| SynCodeBackbone=Llama2-7B-Chat-HF, Decoding Strategy=MCTS, Prompt Engineering=False2026.05 | 8 | 8 | 4 | 183.5 | 165.9 | |
| SynCodeBackbone=Gemma2-2B, Decoding Strategy=Greedy, Prompt Engineering=True2026.05 | 6 | 6 | 1 | — | — | |
| SynCodeBackbone=Llama2-7B-Chat-HF, Decoding Strategy=BS, Prompt Engineering=False2026.05 | 6 | 6 | 4 | 58.7 | 41.1 | |
| XGrammarBackbone=Gemma2-2B, Decoding Strategy=Greedy, Prompt Engineering=False2026.05 | 5 | 5 | 3 | 22.1 | 0.3 | |
| XGrammarBackbone=Gemma2-2B, Decoding Strategy=MCTS, Prompt Engineering=False2026.05 | 5 | 5 | 2 | 293.3 | 271.5 | |
| XGrammarBackbone=Llama2-7B-Chat-HF, Decoding Strategy=BS, Prompt Engineering=False2026.05 | 5 | 3 | 2 | 32.5 | 14.9 | |
| OutlinesBackbone=Gemma2-2B, Decoding Strategy=BS, Prompt Engineering=False2026.05 | 4 | 4 | 2 | 4,347.8 | 4,326 | |
| SynCodeBackbone=Gemma2-2B, Decoding Strategy=Greedy, Prompt Engineering=False2026.05 | 4 | 3 | 0 | 23.5 | 1.7 | |
| SynCodeBackbone=Gemma2-2B, Decoding Strategy=MCTS, Prompt Engineering=False2026.05 | 4 | 4 | 0 | 438.6 | 416.8 | |
| No constraintBackbone=Llama2-7B-Chat-HF, Decoding Strategy=Greedy, Prompt Engineering=False2026.05 | 2 | 2 | 0 | 17.6 | — | |
| No constraintBackbone=Llama2-7B-Chat-HF, Decoding Strategy=Greedy, Prompt Engineering=True2026.05 | 2 | 2 | 0 | — | — | |
| No constraintBackbone=Gemma2-2B, Decoding Strategy=Greedy, Prompt Engineering=False2026.05 | 1 | 1 | 0 | 21.8 | — | |
| SynCodeBackbone=Gemma2-2B, Decoding Strategy=BS, Prompt Engineering=False2026.05 | 1 | 1 | 0 | 54 | 32.2 | |
| XGrammarBackbone=Gemma2-2B, Decoding Strategy=BS, Prompt Engineering=False2026.05 | 1 | 1 | 0 | 34.3 | 12.5 |