Synthetic Grammar Synthesis a^n b^n c^n
100AccuracySEM-CTRL
Evaluation Results
| Method | Links | |
|---|---|---|
| SEM-CTRLAlgorithm=SEM-CTRL, Model=Llama 1B2025.03 | 100 | |
| SEM-CTRLAlgorithm=SEM-CTRL, Model=Llama 70B2025.03 | 100 | |
| o4-miniAlgorithm=API, Model=o4-mini2025.03 | 100 | |
| SEM-CTRLAlgorithm=SEM-CTRL, Model=Llama 1B2025.03 | 100 | |
| SEM-CTRLAlgorithm=SEM-CTRL, Model=Llama 70B2025.03 | 100 | |
| Llama 1BG=GASG2026.04 | 100 | |
| Llama 1BG=G^ ASG2026.04 | 100 | |
| Llama 70BG=GASG2026.04 | 100 | |
| Llama 70BG=G^ ASG2026.04 | 100 | |
| o1-previewAlgorithm=API, Model=o1-preview2025.03 | 96.7 | |
| DeepSeek-R1Algorithm=API, Model=DeepSeek-R12025.03 | 96.7 | |
| o4-miniAlgorithm=API, Model=o4-mini2025.03 | 93.3 | |
| o4 miniG=-2026.04 | 90 | |
| BoNAlgorithm=BoN, Model=Llama 70B2025.03 | 88.9 | |
| o1G=-2026.04 | 86.7 | |
| o1-previewAlgorithm=API, Model=o1-preview2025.03 | 83.3 | |
| DeepSeek-R1Algorithm=API, Model=DeepSeek-R12025.03 | 83.3 | |
| DeepSeek-R1G=-2026.04 | 80 | |
| Llama 70BG=-2026.04 | 76.7 | |
| BoNAlgorithm=BoN, Model=Llama 70B2025.03 | 71.1 | |
| GPT 4.1G=-2026.04 | 63.3 | |
| o3 miniG=-2026.04 | 63.3 | |
| BaseAlgorithm=Base, Model=Llama 70B2025.03 | 60 | |
| BoNAlgorithm=BoN, Model=Llama 1B2025.03 | 48.9 | |
| BaseAlgorithm=Base, Model=Llama 70B2025.03 | 30 | |
| Llama 1BG=-2026.04 | 20 | |
| BaseAlgorithm=Base, Model=Llama 1B2025.03 | 10 | |
| BoNAlgorithm=BoN, Model=Llama 1B2025.03 | 7.8 | |
| BaseAlgorithm=Base, Model=Llama 1B2025.03 | 3.3 |