Systematic Generalization on SCAN Around Right (test)
95.7AccuracyEquiLSTM
Evaluation Results
| Method | Links | |
|---|---|---|
| EquiLSTMGroup=Dir., Training Iterations=10K, Evaluation Protocol=fine-tuned2023.05 | 95.7 | |
| EquiZeroLSTMGroup=Dir., Training Iterations=10K, Evaluation Protocol=fine-tuned2023.05 | 92.5 | |
| EquiGRUGroup=Dir., Fine-tuning iterations=10K2023.05 | 91.6 | |
| G-LSTMGroup=Dir., Training Iterations=200K, Evaluation Protocol=trained2023.05 | 89.6 | |
| G-RNNGroup=Dir., Training iterations=200K iterations2023.05 | 84.5 | |
| G-GRUGroup=Dir., Training iterations=200K2023.05 | 82.7 | |
| EquizeroLSTMGroup=Direction, Zero-shot=true, Iterations=200K2023.05 | 81.7 | |
| EquiRNNGroup=Dir., Training iterations=10K iterations fine-tuning2023.05 | 78.4 | |
| EquiZeroGRUGroup=Dir., Fine-tuning iterations=10K2023.05 | 74.6 | |
| EquiZeroRNNGroup=Dir., Training iterations=10K iterations fine-tuning2023.05 | 64.3 | |
| EquiLSTMGroup=Direction, Zero-shot=true, Iterations=200K2023.05 | 28.2 | |
| RNNGroup=-, Training iterations=200K iterations2023.05 | 5.9 | |
| LSTMGroup=–, Training Iterations=200K, Evaluation Protocol=trained2023.05 | 0.4 | |
| LSTMGroup=–, Zero-shot=true, Iterations=200K2023.05 | 0.4 | |
| GRUGroup=–, Training iterations=200K2023.05 | 0.1 |