Question Answering on ScanQA v1.0 (val) benchmark leaderboard · SOTA2 Research

Question Answering on ScanQA v1.0 (val)

22.59EM@1OracleImage+MCAN (real)

Evaluation Results

Method
OracleImage+MCAN (real)2021.12	22.59	49.43	26.58	18.32	15.37	8.5	33.23	12.45	63.44	12.56
OracleImage+Oscar (mesh)2021.12	22.27	46.59	23.01	13.96	14.23	0	31.37	11.53	57.98	11.11
OracleImage+Oscar (real)2021.12	21.39	45.05	24.29	14.19	14.21	0.67	31.24	11.33	58.23	10.51
ScanQA (multiple)2021.12	21.05	51.23	30.24	20.4	15.11	10.08	33.33	13.14	64.86	13.43
OracleImage+MCAN (mesh)2021.12	20.66	48.04	24.35	17	14.23	0	30.34	11.23	57.01	10.15
ScanQA (single)2021.12	20.28	50.01	29.47	19.84	14.65	9.55	32.37	12.6	61.66	11.86
RandomImage+Oscar (real)2021.12	19.38	46.37	22.91	14.52	11.2	0.56	28.7	10.66	52.86	8.91
RandomImage+MCAN (real)2021.12	19.19	48.15	23.71	15.41	11.81	0	28.9	10.92	53.83	9.35
RandomImage+MCAN (mesh)2021.12	18.59	47.81	22.12	14.49	11.72	7.69	27.61	10.32	50.93	8.37
ScanRefer+MCAN (e2e)2021.12	18.59	46.76	26.93	16.59	11.59	7.87	30.03	11.52	55.41	11.28
RandomImage+Oscar (mesh)2021.12	17.97	43.55	20.67	12.01	11.51	0.57	26.51	9.82	47.82	7.75
VoteNet+MCAN2021.12	17.33	45.54	28.09	16.72	10.75	6.24	29.84	11.41	54.68	10.65
TopDownImage+Oscar2021.12	17.2	43.81	19.75	11.21	15.31	0.71	25.39	9.43	45.21	7.32
ScanRefer+MCAN (pipeline, mesh)2021.12	14.57	43.27	16.71	9.71	13.62	0.64	21.82	8.32	38.35	6.49
ScanRefer+MCAN (pipeline, real)2021.12	14.37	44.12	17.02	10.17	15.77	0.72	22.02	8.45	38.73	6.66
TopDownImage+MCAN2021.12	12.71	41.5	14.82	8.21	16.54	0.74	19.33	7.57	33.14	5.72