Semantic Parsing on SLOG (Generalization Set)
100PP Recursion (Depth 3)NCA
Evaluation results
| Method | Links | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NCANumber of Seeds=10, Evaluation Protocol=type exact match2026.04 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 0 | 100 | 4.7 | 100 | 99.2 | 99.4 | 0 | 0 | 41.4 | 0 | 67.3 | |
| AM-ParserNumber of Seeds=5, Evaluation Protocol=reformatted exact match2026.04 | 100 | 100 | 100 | 100 | 100 | 99.5 | 90.4 | 57.6 | 74.4 | 55.8 | 0 | 99.8 | 100 | 29.4 | 41.4 | 55.6 | 0 | 70.8 | |
| AM-ParserEncoder=BERT-base, Seeds=5, Protocol=reformatted exact match2026.07 | 100 | 100 | 100 | 100 | 100 | 99.5 | 90.4 | 57.6 | 74.4 | 55.8 | 0 | 99.8 | 100 | 29.4 | 41.4 | 55.6 | 0 | 70.8 | |
| Ours (DeBERTa)Encoder=DeBERTa-v3-large, Seeds=10, Protocol=LF exact match2026.07 | 99.1 | 97.8 | 81.9 | 64.8 | 100 | 99.3 | 99.3 | 94.5 | 95.7 | 52 | 97.8 | 91.9 | 99.2 | 99.6 | 99.7 | 84.5 | 84.7 | 90.7 | |
| LLaMANumber of Seeds=5, Evaluation Protocol=reformatted exact match2026.04 | 98.9 | 20.6 | 98.1 | 12.1 | 50.7 | 0 | 55 | 28.9 | 55 | 29.5 | 2.5 | 93.3 | 15.3 | 8.6 | 73.5 | 20.8 | 3 | 40.1 | |
| LLaMASeeds=5, Protocol=reformatted exact match2026.07 | 98.9 | 20.6 | 98.1 | 12.1 | 50.7 | 0 | 71.2 | 28.9 | 55 | 29.5 | 2.5 | 93.3 | 15.3 | 8.6 | 73.5 | 20.8 | 3 | 40.1 | |
| Vanilla TFNumber of Seeds=5, Evaluation Protocol=reformatted exact match, Training Protocol=trained from scratch2026.04 | 98.7 | 13.1 | 55.2 | 0.9 | 0 | 0 | 42.5 | 0 | 34.8 | 0 | 4.7 | 97.1 | 31.9 | 16 | 42.4 | 17.8 | 4.9 | 27.1 | |
| Ours (BERT)Encoder=BERT-base, Seeds=10, Protocol=LF exact match2026.07 | 96.8 | 94.3 | 18.8 | 17.7 | 99.7 | 80.6 | 97.6 | 92.8 | 78.4 | 61.9 | 96.9 | 77.8 | 85.9 | 96.3 | 71.4 | 73.9 | 48.4 | 75.9 | |
| T5Number of Seeds=5, Evaluation Protocol=reformatted exact match2026.04 | 93.1 | 16.6 | 60.9 | 5.3 | 64.1 | 0 | 53.8 | 0.8 | 36.6 | 0.2 | 0 | 98.1 | 100 | 98.5 | 0.4 | 36.8 | 24.9 | 40.6 | |
| T5Seeds=5, Protocol=reformatted exact match2026.07 | 93.1 | 16.6 | 60.9 | 5.3 | 64.1 | 0 | 53.8 | 0.8 | 36.6 | 0.2 | 0 | 98.1 | 100 | 98.5 | 0.4 | 36.8 | 24.9 | 40.6 |