Synthetic Grammar Synthesis on Synthetic Grammar Synthesis (a^m b^n c^m d^n)
100AccuracySEM-CTRL
Evaluation results
| Method | Links | |
|---|---|---|
| SEM-CTRLAlgorithm=SEM-CTRL, Model=Llama 1B2025.03 | 100 | |
| SEM-CTRLAlgorithm=SEM-CTRL, Model=Llama 70B2025.03 | 100 | |
| o4-miniAlgorithm=API, Model=o4-mini2025.03 | 93.3 | |
| o1-previewAlgorithm=API, Model=o1-preview2025.03 | 80 | |
| DeepSeek-R1Algorithm=API, Model=DeepSeek-R12025.03 | 70 | |
| BoNAlgorithm=BoN, Model=Llama 70B2025.03 | 22.2 | |
| BoNAlgorithm=BoN, Model=Llama 1B2025.03 | 1.1 | |
| BaseAlgorithm=Base, Model=Llama 1B2025.03 | 0 | |
| BaseAlgorithm=Base, Model=Llama 70B2025.03 | 0 |