Natural language to strategic temporal logic translation on NL-to-ATL translation (test)
85.6Semantic Accuracygpt-5.4
Evaluation Results
| Method | Links | |
|---|---|---|
| gpt-5.4Size=API, Training=Baseline, Prompting=Few-shot2026.06 | 85.6 | |
| qwen-coder-7bSize=7B, Training=Fine-tuned, Prompting=Few-shot2026.06 | 84.4 | |
| gpt-4.1Size=API, Training=Baseline, Prompting=Few-shot2026.06 | 84.3 | |
| qwen-coder-7bSize=7B, Training=Fine-tuned, Prompting=Zero-shot2026.06 | 83.9 | |
| phi3Size=3.8B, Training=Fine-tuned, Prompting=Few-shot2026.06 | 81.8 | |
| qwen-3bSize=3B, Training=Fine-tuned, Prompting=Few-shot2026.06 | 80.1 | |
| phi3Size=3.8B, Training=Fine-tuned, Prompting=Zero-shot2026.06 | 80 | |
| qwen-3bSize=3B, Training=Fine-tuned, Prompting=Zero-shot2026.06 | 77.8 | |
| mistral-7bSize=7B, Training=Fine-tuned, Prompting=Few-shot2026.06 | 76.6 | |
| mistral-7bSize=7B, Training=Fine-tuned, Prompting=Zero-shot2026.06 | 76.2 | |
| gpt-5.4Size=API, Training=Baseline, Prompting=Zero-shot2026.06 | 69.4 | |
| gpt-4.1Size=API, Training=Baseline, Prompting=Zero-shot2026.06 | 61.1 | |
| qwen-coder-7bSize=7B, Training=Baseline, Prompting=Few-shot2026.06 | 46.3 | |
| phi3Size=3.8B, Training=Baseline, Prompting=Few-shot2026.06 | 41.7 | |
| qwen-3bSize=3B, Training=Baseline, Prompting=Few-shot2026.06 | 36.7 | |
| mistral-7bSize=7B, Training=Baseline, Prompting=Few-shot2026.06 | 34.9 | |
| qwen-coder-7bSize=7B, Training=Baseline, Prompting=Zero-shot2026.06 | 29.8 | |
| phi3Size=3.8B, Training=Baseline, Prompting=Zero-shot2026.06 | 27.5 | |
| mistral-7bSize=7B, Training=Baseline, Prompting=Zero-shot2026.06 | 23.9 | |
| qwen-3bSize=3B, Training=Baseline, Prompting=Zero-shot2026.06 | 15.1 |