Multimodal Science Question Answering on ScienceQA v1.0 (test)

97.47Accuracy (Natural Language Component)MC-CoT_F-Large

Evaluation results

Method
MC-CoT_F-Large2023.11	97.47	90.44	93.18	96.97	93.75	94.49	95.3	94.13	94.88
Mutimodal-CoT_Large2023.11	95.91	82	90.82	95.26	88.8	92.89	92.44	90.31	91.68
MC-CoT_Large2023.11	95.47	89.99	91.82	95.11	92.66	93.24	94.27	91.76	93.37
MC-CoT_F-Base2023.11	93.56	83.58	90.73	94.13	89.24	90.94	90.93	90.38	90.73
MC-CoT_Base2023.11	91.87	84.59	93	92.28	88.3	92.75	90.64	90.64	90.64
LLaVa + GPT-42023.11	91.56	96.74	91.09	90.62	88.99	93.52	92.73	92.16	92.53
LLaVa2023.11	90.36	95.95	88	89.49	88	90.66	90.93	90.9	90.92
LaVIN-13B2023.11	90.32	94.38	87.73	89.44	87.65	90.31	91.19	89.26	90.5
Human2023.11	90.23	84.97	87.48	89.6	87.5	88.1	91.59	82.42	88.4
Chameleon + GPT-42023.11	89.83	74.13	89.82	88.27	77.64	92.13	88.03	83.72	86.54
LLaMA-SciTune_Large2023.11	89.3	95.61	87	93.08	86.67	91.75	84.37	91.3	90.03
LaVIN-7B2023.11	89.25	94.94	85.24	88.51	87.46	88.08	90.16	88.07	89.41
Mutimodal-CoT_Base2023.11	87.52	77.17	85.82	87.88	82.9	86.83	84.65	85.37	84.91
GPT-4 w/ CoT2023.11	85.48	72.44	90.27	82.65	71.49	92.89	86.66	79.04	83.99
LLaMA-SciTune_Base2023.11	84.5	94.15	82.91	88.35	83.64	88.74	85.05	85.6	86.11
LLaMA-Adapter2023.11	84.37	88.3	84.36	83.72	80.32	86.9	85.83	84.05	85.19
Chameleon + ChatGPT2023.11	81.62	70.64	84	79.77	70.8	86.62	81.86	76.53	79.93
LLaMA-Adapter (T)2023.11	79	73.79	80.55	78.3	70.35	83.14	79.77	75.68	78.31
ChatGPT w/ CoT2023.11	78.82	70.98	83.18	77.37	67.92	86.13	80.72	74.03	78.31
GPT-3.5 w/ CoT2023.11	75.44	70.87	78.09	74.68	67.43	79.93	78.23	69.68	75.17
GPT-3.52023.11	74.64	69.74	76	74.44	67.28	77.42	76.8	68.89	73.97
UnifiedQA_Base w/ COT2023.11	71	76.04	78.91	66.42	66.53	81.81	77.06	68.82	74.11
UnifiedQA_Base2023.11	68.16	69.18	74.91	63.78	61.38	77.84	72.98	65	70.12
Patch-TRM2023.11	65.19	46.79	65.55	66.96	55.28	64.95	58.04	67.5	61.42
DFAF2023.11	64.03	48.82	63.55	65.88	54.49	64.11	57.12	67.17	60.72
BAN2023.11	60.88	46.57	66.64	62.61	52.6	65.51	56.83	63.94	59.37
VILT2023.11	60.48	63.89	60.27	63.2	61.38	57	60.72	61.9	61.14
Top-Down2023.11	59.5	54.33	61.82	62.9	54.88	59.79	57.27	62.16	59.02
VisualBERT2023.11	59.33	69.18	61.18	62.71	62.17	58.54	62.96	59.92	61.87
MCAN2023.11	56.08	46.23	58.09	59.43	51.17	55.4	51.65	59.72	54.54
Random Choice2023.11	40.28	46.13	29.25	47.75	40.08	33.66	39.35	40.67	39.83