Question Answering on bAbI (test)
0.032Mean ErrorStrongly Supervised MemNN
Evaluation results
| Method | Links | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Strongly Supervised MemNNSupervision=Strongly Supervised2015.03 | 0.032 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2 | — | — | — | |
| Strongly Supervised MemNNSupervision=Strong2015.03 | 0.032 | 0 | 0 | 0 | 0 | 0.3 | 0 | 3.3 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 24.6 | 2.1 | 31.9 | 0 | 2 | — | — | — | |
| MemN2N PE LS RN*Representation=PE, Linear Start (LS)=true, Time Index Noise (RN)=true, Embedding dimension (d)=100, Non-linearity=ReLU2015.03 | 0.042 | 0 | 0.3 | 2.1 | 0 | 0.8 | 0.1 | 2 | 0.9 | 0.3 | 0 | 0.1 | 0 | 0 | 0.1 | 0 | 51.8 | 18.6 | 5.3 | 2.3 | 0 | 3 | — | — | — | |
| MemN2NHops=1, Sentence Representation=PE, Linear Start=true, Random Noise=true, Joint Training=false2015.03 | 0.066 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4 | — | — | — | |
| MemN2N PE LS RNRepresentation=PE, Linear Start (LS)=true, Time Index Noise (RN)=true2015.03 | 0.066 | 0 | 0.3 | 9.3 | 0 | 0.8 | 0 | 3.7 | 0.8 | 0.8 | 2.4 | 0 | 0 | 0 | 0 | 0 | 0.4 | 40.7 | 6.7 | 66.5 | 0 | 4 | — | — | — | |
| MemN2NHops=1, Sentence Representation=PE, Linear Start=true, Joint Training=false2015.03 | 0.072 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4 | — | — | — | |
| MemN2N PE LSRepresentation=PE, Linear Start (LS)=true2015.03 | 0.072 | 0 | 0.5 | 15 | 0 | 0.6 | 0.1 | 3.2 | 2.2 | 2 | 3.3 | 0 | 0 | 0 | 0 | 0 | 0.1 | 41.1 | 8.6 | 66.7 | 0 | 4 | — | — | — | |
| MemN2NHops=3, Sentence Representation=PE, Linear Start=true, Random Noise=true, Joint Training=true, Variation=Config B2015.03 | 0.075 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 6 | — | — | — | |
| MemN2N PE JointRepresentation=PE, Training=Joint2015.03 | 0.075 | 0 | 1 | 6.8 | 0 | 6.1 | 0.1 | 6.6 | 2.7 | 0 | 0.5 | 0 | 0.1 | 0 | 0 | 0.2 | 0.2 | 41.8 | 8 | 75.7 | 0 | 6 | — | — | — | |
| MemN2NHops=3, Sentence Representation=PE, Linear Start=true, Random Noise=true, Joint Training=true, Variation=Config A2015.03 | 0.079 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 6 | — | — | — | |
| MemN2N 3 hops JointHops=3, Training=Joint2015.03 | 0.079 | 0 | 2.3 | 14 | 0 | 7.5 | 0.2 | 6.1 | 4 | 0 | 0.4 | 0 | 0 | 0 | 0.2 | 0 | 0.4 | 41.7 | 8.6 | 73.3 | 0 | 6 | — | — | — | |
| MemN2NHops=1, Sentence Representation=PE, Joint Training=false2015.03 | 0.094 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 6 | — | — | — | |
| MemN2N PERepresentation=PE2015.03 | 0.094 | 0 | 0.4 | 12.6 | 0 | 0.8 | 0.2 | 5.7 | 2.4 | 1.3 | 1.7 | 0 | 0 | 0.1 | 0.2 | 0 | 48.6 | 40.3 | 7.4 | 66.6 | 0 | 6 | — | — | — | |
| MemN2NHops=3, Sentence Representation=PE, Linear Start=true, Joint Training=true2015.03 | 0.109 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 7 | — | — | — | |
| MemN2N 2 hops JointHops=2, Training=Joint2015.03 | 0.109 | 0 | 1.3 | 15.8 | 0 | 7.2 | 0.7 | 10.5 | 4.7 | 0.4 | 0.6 | 0 | 0 | 0 | 0.2 | 0 | 46.4 | 39.7 | 10.1 | 80.8 | 0 | 7 | — | — | — | |
| MemN2NHops=3, Sentence Representation=PE, Linear Start=true, Layer-wise Weight Tying=true, Joint Training=true2015.03 | 0.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 6 | — | — | — | |
| MemN2N PE LS LW JointRepresentation=PE, Linear Start (LS)=true, Layer-wise Weight Tying (LW)=RNN-style, Training=Joint2015.03 | 0.11 | 0 | 0.8 | 18.3 | 0 | 0.8 | 0.1 | 8.4 | 1.4 | 0.2 | 0 | 0.4 | 0 | 0 | 1.7 | 0 | 49.2 | 40 | 8.4 | 89.5 | 0 | 6 | — | — | — | |
| MemN2NHops=1, Sentence Representation=BoW, Joint Training=false2015.03 | 0.154 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 9 | — | — | — | |
| MemN2N BoWRepresentation=BoW2015.03 | 0.154 | 0 | 0.6 | 17.8 | 31.8 | 14.2 | 0.1 | 10.7 | 1.4 | 1.8 | 1.9 | 0 | 0 | 0 | 0 | 12.5 | 50.9 | 47.4 | 41.3 | 75.4 | 0 | 9 | — | — | — | |
| MemN2NHops=2, Sentence Representation=PE, Linear Start=true, Joint Training=true2015.03 | 0.245 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 16 | — | — | — | |
| MemN2N 1 hop JointHops=1, Training=Joint2015.03 | 0.245 | 0 | 62 | 80 | 21.4 | 8.7 | 6.1 | 14.8 | 8.9 | 3.7 | 10.3 | 8.3 | 0 | 5.6 | 30.9 | 42.6 | 47.3 | 40 | 9.2 | 91 | 0 | 16 | — | — | — | |
| LSTM2015.03 | 0.364 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 16 | — | — | — | |
| LSTMArchitecture=LSTM2015.03 | 0.364 | 0 | 81.9 | 83.1 | 0.2 | 1.2 | 51.8 | 24.9 | 34.1 | 20.2 | 30.1 | 10.3 | 23.4 | 6.1 | 81 | 78.7 | 51.9 | 50.1 | 6.8 | 90.3 | 2.1 | 16 | — | — | — | |
| STMruns=102020.02 | 0.39 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.15 | |
| MemNN WSH2015.03 | 0.392 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 17 | — | — | — | |
| MemN2N WSHRepresentation=WSH2015.03 | 0.392 | 0.1 | 39.6 | 79.5 | 36.6 | 21.1 | 49.9 | 35.1 | 42.7 | 36.4 | 76 | 25.3 | 0 | 12.3 | 8.7 | 68.8 | 50.9 | 51.1 | 45.8 | 100 | 4.1 | 17 | — | — | — | |
| MNM-pruns=102020.02 | 0.55 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.18 | |
| UTruns=102020.02 | 1.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.21 | |
| TPRruns=102020.02 | 1.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.81 | |
| Spiking RelNet2022.05 | 1.5 | 1 | — | — | 0.1 | 2.3 | 0.4 | 1.4 | 0.9 | 0.7 | 1.7 | 2.1 | 4.2 | 3.6 | 0 | 0 | — | 2.3 | 0.2 | 3.7 | 0.4 | 3 | — | — | — | |
| DMN+Controller type=GRU2016.06 | 2.81 | 0 | 0.3 | 1.1 | 0 | 0.5 | 0 | 2.4 | 0 | 0 | 0 | 0 | 0 | 0 | 0.2 | 0 | 45.3 | 4.2 | 2.1 | 0 | 0 | — | — | — | — | |
| MemN2NController type=GRU2016.06 | 4.24 | 0 | 0.3 | 2.1 | 0 | 0.8 | 0.1 | 2 | 0.9 | 0.3 | 0 | 0.1 | 0 | 0 | 0.1 | 0 | 51.8 | 18.6 | 5.3 | 2.3 | 0 | — | — | — | — | |
| NUTMcore=DNC core, p=42019.05 | 5.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.9 | — | |
| NUTMruns=102020.02 | 5.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.3 | |
| ADNC2019.05 | 6.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.7 | — | |
| SDNC2019.05 | 6.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.5 | — | |
| NUTMcore=DNC core, p=22019.05 | 7.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.6 | — | |
| DNC-MD2019.05 | 9.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.6 | — | |
| NUTMcore=DNC core, p=12019.05 | 9.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.5 | — | |
| H-Memmem. acc.=3-hop2022.05 | 10 | 0 | 0.2 | 26.9 | 0 | 1.3 | 1.2 | 0.8 | 0.5 | 3.3 | 1.5 | 0 | 0 | 0 | 1.1 | 0 | 54.8 | 28.7 | 1.9 | 77.1 | 0 | 4 | — | — | — | |
| Hebbian-enriched SNNfb. delay=30 ms2022.05 | 10.9 | 0 | 3.3 | 59.2 | 0 | 2.1 | 4.2 | 3.4 | 0.5 | 2.7 | 3.9 | 0 | 0.1 | 0 | 2.7 | 0 | 53.4 | 15.7 | 1.8 | 65.8 | 0 | 4 | — | — | — | |
| DNCruns=102020.02 | 12.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.8 | |
| Hebbian-enriched SNNfb. delay=1 ms2022.05 | 13.9 | 0 | 10.4 | 58.6 | 0 | 1.4 | 28.2 | 3.2 | 0.8 | 1.4 | 5.8 | 0 | 0 | 0.3 | 4.4 | 0 | 54.1 | 37.7 | 0.8 | 66.5 | 0 | 7 | — | — | — | |
| DNC2019.05 | 16.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 7.6 | — | |
| H-Memmem. acc.=1-hop2022.05 | 18 | 0 | 64.2 | 58.6 | 0 | 4.1 | 12.2 | 0.8 | 0.4 | 15.5 | 21.3 | 0.1 | 0 | 2.3 | 7.9 | 1 | 54.2 | 38.8 | 4.8 | 74.7 | 0 | 9 | — | — | — | |
| D-NTMNumber of steps=3, Addressing mechanism=Discrete, Controller type=GRU2016.06 | 21.79 | 0 | 62.29 | 41.45 | 0 | 1.45 | 11.04 | 5.62 | 0.74 | 32.5 | 20.83 | 16.87 | 4.58 | 5 | 60.2 | 40.26 | 45.41 | 9.16 | 1.66 | 76.66 | 0 | — | — | — | — | |
| D-NTMNumber of steps=3, Addressing mechanism=Soft, Controller type=GRU2016.06 | 24.24 | 0 | 46.66 | 47.08 | 0 | 1.25 | 20.62 | 7.29 | 11.02 | 39.37 | 20 | 30.62 | 5.41 | 7.91 | 58.12 | 36.04 | 46.04 | 21.25 | 6.87 | 75.88 | 3.33 | — | — | — | — | |
| D-NTMNumber of steps=1, Addressing mechanism=Discrete, Controller type=GRU2016.06 | 27.93 | 6.66 | 56.04 | 72.08 | 0 | 1.04 | 44.79 | 19.58 | 18.46 | 34.37 | 50.83 | 4.16 | 6.66 | 2.29 | 63.75 | 39.27 | 51.35 | 16.04 | 3.54 | 64.63 | 3.12 | — | — | — | — | |
| D-NTMNumber of steps=1, Addressing mechanism=Soft, Controller type=GRU2016.06 | 29.51 | 5.41 | 58.54 | 74.58 | 0 | 1.66 | 40.2 | 19.16 | 12.58 | 36.66 | 52.29 | 31.45 | 7.7 | 5.62 | 60 | 36.87 | 49.16 | 17.91 | 3.95 | 73.74 | 2.7 | — | — | — | — | |
| NTMNumber of steps=1, Addressing mechanism=LBA*, Controller type=GRU2016.06 | 31.42 | 16.3 | 57.08 | 74.16 | 0 | 1.46 | 23.33 | 21.67 | 25.76 | 24.79 | 41.46 | 18.96 | 25.83 | 6.67 | 58.54 | 36.46 | 71.15 | 43.75 | 3.96 | 75.89 | 1.25 | — | — | — | — | |
| NTMNumber of steps=3, Addressing mechanism=CBA, Controller type=GRU2016.06 | 32.76 | 0 | 59.38 | 65.21 | 0 | 1.46 | 54.8 | 37.7 | 8.82 | 0 | 23.75 | 0.28 | 23.75 | 83.13 | 57.71 | 21.88 | 50 | 56.25 | 47.5 | 63.65 | 0 | — | — | — | — | |
| NTMNumber of steps=3, Addressing mechanism=LBA*, Controller type=GRU2016.06 | 32.85 | 0 | 61.67 | 83.54 | 0 | 0.83 | 48.13 | 7.92 | 25.38 | 37.8 | 56.25 | 3.96 | 28.75 | 5.83 | 61.88 | 35.62 | 46.15 | 43.75 | 47.5 | 61.56 | 0.4 | — | — | — | — | |
| NTMNumber of steps=1, Addressing mechanism=CBA, Controller type=GRU2016.06 | 33.6 | 16.88 | 55.7 | 55 | 0 | 20.41 | 21.04 | 21.67 | 21.05 | 24.17 | 33.13 | 31.88 | 30 | 5.63 | 59.17 | 42.3 | 71.15 | 43.75 | 47.5 | 71.51 | 0 | — | — | — | — | |
| LSTMController type=GRU2016.06 | 36.41 | 0 | 81.9 | 83.1 | 0.2 | 1.2 | 51.8 | 24.9 | 34.1 | 20.2 | 30.1 | 10.3 | 23.4 | 6.1 | 81 | 78.7 | 51.9 | 50.1 | 6.8 | 90.3 | 2.1 | — | — | — | — | |
| baseline (Weston et al.)2017.06 | — | 50 | 20 | 20 | 61 | 70 | 48 | 49 | 45 | 64 | 44 | 72 | 74 | 94 | 27 | 21 | 23 | 51 | 52 | 8 | 91 | — | 0.492 | — | — | |
| EURNNArchitecture=one-way RNN, Memory mechanism=None, Attention mechanism=None2017.06 | — | 47.2 | 24.3 | 22.5 | 56.1 | 56.2 | 50.5 | 71.9 | 56.5 | 60.6 | 42.6 | 72.1 | 72.7 | 92.4 | 20 | 25 | 43.3 | 51.2 | 89.7 | 9 | 93.3 | — | 0.529 | — | — | |
| GORUArchitecture=one-way RNN, Memory mechanism=None, Attention mechanism=None2017.06 | — | 45.8 | 39.5 | 33.5 | 62.7 | 87 | 53.6 | 77.7 | 75 | 62.9 | 45.4 | 69.3 | 69.9 | 92.7 | 37.9 | 55.2 | 44 | 59.6 | 90.5 | 8.9 | 97.7 | — | 0.604 | — | — | |
| GORU2017.10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 60 | — | — | |
| GRUArchitecture=one-way RNN, Memory mechanism=None, Attention mechanism=None2017.06 | — | 49.1 | 38.5 | 32.2 | 64.6 | 78 | 50.5 | 79.5 | 75.5 | 63.9 | 44.8 | 71.2 | 71.6 | 94.2 | 39.2 | 57.4 | 45.9 | 50.5 | 89.9 | 9.6 | 97.7 | — | 0.582 | — | — | |
| LSTMArchitecture=one-way RNN, Memory mechanism=None, Attention mechanism=None2017.06 | — | 49.3 | 32.3 | 20.6 | 67.5 | 52.3 | 49.3 | 76.9 | 76.8 | 63.5 | 46 | 71.1 | 71.9 | 93.8 | 34.4 | 20.9 | 45.9 | 51.6 | 91.8 | 8.2 | 96.5 | — | 0.56 | — | — | |
| LSTM2017.10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 49 | — | — | |
| MemN2N2017.10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 86 | — | — | |
| RUM2017.10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 73.2 | — | — |