Systematic Generalization on SCAN Around Right (val)
99.8AccuracyEquiLSTM
Evaluation Results
| Method | Links | |
|---|---|---|
| EquiLSTMGroup=Dir., Training Iterations=10K, Evaluation Protocol=fine-tuned2023.05 | 99.8 | |
| EquiGRUGroup=Dir., Fine-tuning iterations=10K2023.05 | 99.4 | |
| LSTMGroup=–, Training Iterations=200K, Evaluation Protocol=trained2023.05 | 98.9 | |
| LSTMGroup=–, Zero-shot=true, Iterations=200K2023.05 | 98.9 | |
| EquizeroLSTMGroup=Direction, Zero-shot=true, Iterations=200K2023.05 | 98.7 | |
| G-LSTMGroup=Dir., Training Iterations=200K, Evaluation Protocol=trained2023.05 | 98.4 | |
| EquiZeroLSTMGroup=Dir., Training Iterations=10K, Evaluation Protocol=fine-tuned2023.05 | 98.2 | |
| GRUGroup=–, Training iterations=200K2023.05 | 97.9 | |
| EquiRNNGroup=Dir., Training iterations=10K iterations fine-tuning2023.05 | 97.7 | |
| G-GRUGroup=Dir., Training iterations=200K2023.05 | 97.1 | |
| G-RNNGroup=Dir., Training iterations=200K iterations2023.05 | 96.6 | |
| RNNGroup=-, Training iterations=200K iterations2023.05 | 94.9 | |
| EquiZeroGRUGroup=Dir., Fine-tuning iterations=10K2023.05 | 93.6 | |
| EquiZeroRNNGroup=Dir., Training iterations=10K iterations fine-tuning2023.05 | 89.5 | |
| EquiLSTMGroup=Direction, Zero-shot=true, Iterations=200K2023.05 | 80.5 |