Multi-modal Reasoning on NuPlanQA EVAL benchmark leaderboard · SOTA2 Research

Multi-modal Reasoning on NuPlanQA EVAL

68.5Traffic Light AccuracyGPT-4o

Evaluation Results

Method	Links
GPT-4o2024.10		68.5	91.7	95	85.1	79.6	76.5	67.7	74.6	86.8	85.1	82.4	84.8	81.5
GPT-4o2024.10		66.5	92.2	94.4	84.4	81.2	77	68.5	75.6	86.4	86.6	81.4	84.8	81.6
Gemini-1.5-Pro2024.10		64.5	93.5	95.8	84.6	69.1	78.6	73.2	73.6	73.8	73.8	80.4	76	78.1
Gemini-1.5-Pro2024.10		64.5	93.1	95.8	84.5	71.1	79.6	72.7	74.5	79.6	76.7	80.9	79.1	79.4
BEV-LLM2024.10		61.1	89.4	89.6	80	78.5	75.5	68.2	74.1	79.1	83.2	83.4	81.9	78.7
BEV-LLM2024.10		58.1	87.1	84.9	76.7	75.1	75	66.8	72.3	76	81.2	81.4	79.5	76.2
VideoLLaMA2_7B2024.10		54.2	59.4	93.8	69.1	70.7	78.6	66.4	71.9	72.3	82.7	81.4	78.8	73.3
LLaVA-OV_7B2024.10		53.2	89.4	96.4	79.7	77.9	77.6	73.2	74.3	75.4	76.2	82.9	77.5	77.8
LLaVA-OV_7B2024.10		52.2	93.5	96.4	80.7	72.9	74	68.6	71.8	72.3	72.3	79.9	74.8	75.8
Qwen2.5-VL_7B2024.10		51.7	71.9	69.3	64.3	49.2	23	30.9	34.4	30.4	39.6	10.1	26.7	41.8
Qwen2.5-VL_7B2024.10		51.2	70.5	62.5	61.4	47	16.8	23.2	29	27.2	40.6	12.1	26.6	39
VideoLLaMA2_7B2024.10		50.2	64.1	93.8	69.4	71.3	79.6	64.1	71.7	69.6	84.2	81.4	78.4	73.2
LLaVA-NV_7B2024.10		48.8	41.5	63.5	51.3	51.9	50.5	41.8	48.1	56.5	60.4	69.8	62.2	53.9
LLaVA-NV_32B2024.10		47.8	74.7	93.8	72.1	69.1	74	63.6	68.9	72.8	69.3	76.9	73	71.3
LLaVA-NV_32B2024.10		46.8	73.3	91.7	70.6	70.7	71.9	58.2	66.9	71.2	68.8	82.9	74.3	70.6
LLaVA-NV_7B2024.10		44.3	46.5	72.4	54.4	57.5	52	44.1	51.2	57.6	65.8	71.4	64.9	56.8
InternVL-1.5_20B2024.10		40.9	70.5	75.5	62.3	48.6	50	44.5	47.7	57.1	55	60.8	57.6	55.9
InternVL-1.5_20B2024.10		39.4	68.7	72.4	60.2	54.7	48.5	41.4	48.2	62.3	58.9	58.8	60	56.1