Multimodal Numerical Reasoning on GeoQA (test)
92.3Total AccuracyHuman
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| HumanInput modality=Text-Diagram2021.05 | 92.3 | 94.2 | 90.5 | 87 | |
| HumanInput modality=Text-Only2021.05 | 63 | 58 | 71.7 | 55.6 | |
| NGS AuxiliaryInput modality=Text-Diagram, Program-based reasoning=true, Auxiliary supervision=true2021.05 | 60 | 71.5 | 48.8 | 29.6 | |
| NGSInput modality=Text-Diagram, Program-based reasoning=true, Auxiliary supervision=false2021.05 | 57.4 | 68.6 | 43.5 | 44.4 | |
| BERT2ProgInput modality=Text-Only, Program-based reasoning=true2021.05 | 54.4 | 65.7 | 41 | 37 | |
| Seq2ProgInput modality=Text-Only, Program-based reasoning=true2021.05 | 54.2 | 66.4 | 38.5 | 42.6 | |
| Seq2Prog + DiagramInput modality=Text-Diagram, Program-based reasoning=true2021.05 | 53.4 | 62.4 | 44.5 | 31.5 | |
| BERT2Prog + DiagramInput modality=Text-Diagram, Program-based reasoning=true2021.05 | 52.5 | 65.7 | 37.1 | 31.5 | |
| MCANProgram-based reasoning=false, Input modality=Text-Diagram2021.05 | 39.5 | 43.2 | 36 | 29.6 | |
| RNProgram-based reasoning=false, Input modality=Text-Diagram2021.05 | 38.2 | 42.2 | 34.3 | 27.8 | |
| FiLMProgram-based reasoning=false, Input modality=Text-Diagram2021.05 | 32.8 | 33.6 | 32.9 | 25.9 |