Grammar Synthesis on Synthetic Grammar Synthesis
100Accuracy ($a^n b^n c^n$)Llama 1B
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Llama 1BAlgorithm=SEM-CTRL2025.03 | 100 | 100 | 100 | |
| Llama 8BAlgorithm=SEM-CTRL2025.03 | 100 | 100 | 100 | |
| Llama 70BAlgorithm=SEM-CTRL2025.03 | 100 | 100 | 100 | |
| o4-miniAlgorithm=API2025.03 | 93.3 | 93.3 | 100 | |
| DeepSeek-R1Algorithm=API2025.03 | 87.6 | 70 | 99.6 | |
| o1-previewAlgorithm=API2025.03 | 83.3 | 80.6 | 98.6 | |
| Llama 70BAlgorithm=BoN2025.03 | 78.3 | 59.6 | 91.1 | |
| Llama 8BAlgorithm=BoN2025.03 | 39.3 | 33.7 | 98.9 | |
| Llama 70BAlgorithm=Base2025.03 | 30 | 3.1 | 77.4 | |
| Llama 1BAlgorithm=BoN2025.03 | 16 | 5.4 | 67.1 | |
| Llama 8BAlgorithm=Base2025.03 | 7.7 | 0 | 42.5 | |
| Llama 1BAlgorithm=Base2025.03 | 6.1 | 2.1 | 40.5 |