Visual Question Answering on CLEVR (test)
99.8Overall AccuracyNS-VQA
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NS-VQAProgram Annotations=2702018.10 | 99.8 | 99.9 | 99.7 | — | — | — | — | — | — | — | — | — | — | — | 99.9 | 99.8 | 99.8 | |
| NS-VQAexternal program annotations=true2021.04 | 99.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MDETR2021.04 | 99.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OCCAM2021.04 | 99.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NS-CL2019.05 | 99.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TbD+hres-Privileged state-description=true, Spatial tensor=28x282018.08 | 99.1 | 99.2 | 97.6 | — | — | — | — | — | — | — | — | — | — | — | 99.4 | 99.5 | 99.6 | |
| TbD+reg+hresProgram Annotations=700K2018.10 | 99.1 | 99.2 | 97.6 | — | — | — | — | — | — | — | — | — | — | — | 99.4 | 99.5 | 99.6 | |
| TbD-netProgram=supervised, #Modules=39, #Param.=115M2018.12 | 99.1 | 99.2 | 97.6 | — | — | — | — | — | — | — | — | — | — | — | 99.4 | 99.5 | 99.6 | |
| MAC2018.08 | 98.9 | 99.5 | 97.2 | — | — | — | — | — | — | — | — | — | — | — | 99.4 | 99.3 | 99.5 | |
| MAC2018.10 | 98.9 | 99.5 | 97.1 | — | — | — | — | — | — | — | — | — | — | — | 99.1 | 99.5 | 99.5 | |
| MAC2019.05 | 98.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAC2021.04 | 98.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HAN+RN++Input image resolution=224x224, Spatial dimension=14x142018.08 | 98.8 | 99.6 | 97.2 | — | — | — | — | — | — | — | — | — | — | — | 96.9 | 99.6 | 99.6 | |
| TbD-Privileged state-description=true2018.08 | 98.7 | 98.9 | 96.8 | — | — | — | — | — | — | — | — | — | — | — | 99.1 | 99.4 | 99.2 | |
| DDRprog-Privileged state-description=true2018.08 | 98.3 | 98.8 | 96.5 | — | — | — | — | — | — | — | — | — | — | — | 98.4 | 99.1 | 99 | |
| DDRprogProgram Annotations=700K2018.10 | 98.3 | 98.8 | 96.5 | — | — | — | — | — | — | — | — | — | — | — | 98.4 | 99.1 | 99 | |
| single-hop + LCGNLocal features=14 x 14 x 1024 convolutional grid features, Message passing iterations (T)=42019.05 | 97.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XNM-DetProgram=supervised, #Modules=12, #Param.=0.55M2018.12 | 97.7 | 98.7 | 96 | — | — | — | — | — | — | — | — | — | — | — | 98 | 98.4 | 97.6 | |
| FiLM2021.04 | 97.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FiLM2018.08 | 97.6 | 99.2 | 94.5 | — | — | — | — | — | — | — | — | — | — | — | 93.8 | 99.2 | 99 | |
| CNN+GRU+FiLM2018.10 | 97.6 | 99.2 | 94.5 | — | — | — | — | — | — | — | — | — | — | — | 93.8 | 99.2 | 99 | |
| FiLM2019.05 | 97.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MoViebackbone=ResNet-101, training epochs=452020.04 | 97.42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PG + EE (700k)-Privileged state-description=true2018.08 | 96.9 | 97.1 | 92.7 | — | — | — | — | — | — | — | — | — | — | — | 98.7 | 98.1 | 98.9 | |
| HAN+RN+Relational model=Larger relational model2018.08 | 96.9 | 98.6 | 92.8 | — | — | — | — | — | — | — | — | — | — | — | 94.9 | 98.9 | 98.2 | |
| IEPProgram Annotations=700K2018.10 | 96.9 | 97.1 | 92.7 | — | — | — | — | — | — | — | — | — | — | — | 98.7 | 98.1 | 98.9 | |
| PG+EEProgram=supervised, #Modules=39, #Param.=40.4M2018.12 | 96.9 | 97.1 | 92.7 | — | — | — | — | — | — | — | — | — | — | — | 98.7 | 98.1 | 98.9 | |
| StackNMNProgram=supervised, #Modules=9, #Param.=7.32M2018.12 | 96.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hyperbolic RN2018.08 | 95.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RN2018.08 | 95.5 | 97.8 | 90.1 | — | — | — | — | — | — | — | — | — | — | — | 93.6 | 97.9 | 97.1 | |
| CNN+LSTM+RN2018.10 | 95.5 | 97.8 | 90.1 | — | — | — | — | — | — | — | — | — | — | — | 93.6 | 97.9 | 97.1 | |
| RN2019.05 | 95.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HAN+Sum+Relational model=Larger relational model2018.08 | 94.7 | 97.3 | 88.9 | — | — | — | — | — | — | — | — | — | — | — | 88 | 98.1 | 97 | |
| StackNMNProgram=scratch, #Modules=9, #Param.=7.32M2018.12 | 93 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Stack-NMN2019.05 | 93 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Human2018.08 | 92.6 | 96.6 | 86.7 | — | — | — | — | — | — | — | — | — | — | — | 86.5 | 95 | 96 | |
| Humans2018.10 | 92.6 | 96.6 | 86.7 | — | — | — | — | — | — | — | — | — | — | — | 86.4 | 95 | 96 | |
| Human2018.12 | 92.6 | 96.6 | 86.7 | — | — | — | — | — | — | — | — | — | — | — | 86.4 | 95 | 96 | |
| NS-VQAProgram Annotations=1802018.10 | 89.5 | 92.9 | 85 | — | — | — | — | — | — | — | — | — | — | — | 83.4 | 92.2 | 90.6 | |
| Dependency Tree2018.10 | 89.3 | 94.2 | 81.4 | — | — | — | — | — | — | — | — | — | — | — | 81.6 | 90.5 | 97.1 | |
| N2NMNsOptimization Protocol=policy search after cloning2017.04 | 83.7 | 85.7 | 68.5 | 73.8 | 89.7 | 87.7 | 93.1 | 84.8 | 91.5 | 90.6 | 92.6 | 82.8 | 89.6 | 90 | — | — | — | |
| N2NMN2018.08 | 83.7 | 85.7 | 68.5 | — | — | — | — | — | — | — | — | — | — | — | 85 | 90 | 88.9 | |
| N2NMNProgram Annotations=700K2018.10 | 83.7 | 85.7 | 68.5 | — | — | — | — | — | — | — | — | — | — | — | 84.9 | 90 | 88.7 | |
| N2NMNProgram=supervised, #Modules=122018.12 | 83.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| N2NMNsOptimization Protocol=cloning expert2017.04 | 78.9 | 83.3 | 63.3 | 68.2 | 87.2 | 85.4 | 90.5 | 80.2 | 88.9 | 88.3 | 89.4 | 52.5 | 85.4 | 86.7 | — | — | — | |
| LBP-SIG2018.08 | 78 | 79.6 | 61.3 | — | — | — | — | — | — | — | — | — | — | — | 80.7 | 88.6 | 76.3 | |
| SAN*Implementation=from [25]2018.08 | 76.6 | 82.7 | 64.4 | — | — | — | — | — | — | — | — | — | — | — | 77.4 | 82.6 | 75.4 | |
| NS-VQAProgram Annotations=902018.10 | 74.4 | 87.4 | 64.5 | — | — | — | — | — | — | — | — | — | — | — | 53.7 | 79.7 | 77.4 | |
| CNN+LSTM+SAN2018.10 | 73.2 | 77.9 | 59.7 | — | — | — | — | — | — | — | — | — | — | — | 75.1 | 80.9 | 70.8 | |
| single-hopLocal features=14 x 14 x 1024 convolutional grid features2019.05 | 72.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NMNlayout=expert layout2017.04 | 72.1 | 79.3 | 52.5 | 61.2 | 77.9 | 75.2 | 84.2 | 68.9 | 82.6 | 80.2 | 80.7 | 74.4 | 77.6 | 79.3 | — | — | — | |
| N2NMNsOptimization Protocol=policy search from scratch2017.04 | 69 | 72.7 | 55.1 | 71.6 | 85.1 | 79 | 88.1 | 74 | 86.6 | 84.1 | 50.1 | 53.9 | 48.6 | 51.1 | — | — | — | |
| N2NMNProgram=scratch, #Modules=122018.12 | 69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CNN+LSTM+SA2017.04 | 68.5 | 71.1 | 52.2 | 60 | 82 | 74 | 87 | 81 | 88 | 85 | 52 | 55 | 51 | 51 | — | — | — | |
| SANImplementation=from [34]2018.08 | 68.5 | 71.1 | 52.2 | — | — | — | — | — | — | — | — | — | — | — | 73.5 | 85.3 | 52.3 | |
| CNN+LSTM2017.04 | 52.3 | 65.2 | 43.7 | 57 | 72 | 69 | 59 | 32 | 58 | 48 | 51 | 54 | 51 | 53 | — | — | — | |
| CNN+LSTM2018.08 | 52.3 | 65.2 | 43.7 | — | — | — | — | — | — | — | — | — | — | — | 67.1 | 49.3 | 53 | |
| CNN+LSTM+MCB2017.04 | 51.4 | 63.4 | 42.1 | 57 | 71 | 68 | 59 | 32 | 57 | 48 | 51 | 52 | 50 | 51 | — | — | — | |
| CNN+BoW2017.04 | 48.4 | 59.5 | 38.9 | 50 | 54 | 49 | 56 | 32 | 58 | 47 | 52 | 52 | 51 | 52 | — | — | — | |
| LSTM-only2018.08 | 46.8 | 61.1 | 41.7 | — | — | — | — | — | — | — | — | — | — | — | 69.8 | 36.8 | 51.8 | |
| Q-type baseline2018.08 | 41.8 | 50.2 | 34.6 | — | — | — | — | — | — | — | — | — | — | — | 51 | 36 | 51.3 |