Question Answering on SpotterQA

94.4Overall Accuracyclaude-opus-4

Evaluation results

Method
claude-opus-42025.10	94.4	97.7	89.7	89.2	88.3	92.2	78.9	99.8
gpt-52025.10	94	98	88.2	87.8	88.3	90.2	73.7	99.9
claude-sonnet-42025.10	93.9	96.6	90	90.6	88.8	88.2	81.6	99.6
gpt-52025.10	93.9	98	87.9	87.8	88.8	86.3	78.9	99.7
claude-opus-42025.10	93.7	97	88.9	85.9	87.8	92.2	86.8	99.3
claude-sonnet-42025.10	93.7	97	88.9	88.3	88.8	90.2	81.6	99.7
o4-mini2025.10	93	97.7	86.4	84.5	85.4	86.3	78.9	100
o4-mini2025.10	93	96.8	87.7	84.5	87.8	92.2	76.3	99.6
o32025.10	92.8	96.6	87.4	86.4	86.3	88.2	81.6	100
o4-mini2025.10	92.7	95.3	88.9	88.7	87.3	92.2	78.9	100
gpt-52025.10	92.6	96.4	87.1	87.3	86.3	88.2	73.7	100
Human2025.10	92.5	92.8	91.9	96.2	94.1	81.6	86.5	100
o32025.10	92.5	96.6	86.6	87.3	88.8	86.3	76.3	98.2
o4-mini2025.10	92.5	96.6	86.6	86.9	84.9	84.3	76.3	100
gpt-52025.10	92.4	97	85.9	86.9	85.9	86.3	73.7	100
o32025.10	92.4	96.4	86.6	87.3	86.8	88.2	76.3	100
o32025.10	91.4	95.3	85.6	84	86.8	84.3	81.6	97.9
gpt-4.12025.10	90.9	95.9	83.8	81.7	82.9	84.3	76.3	99.9
claude-opus-42025.10	90.6	93.6	86.4	86.9	84.9	84.3	81.6	99.9
claude-sonnet-42025.10	90.4	94.1	85.1	83.1	81	94.1	81.6	99.9
gpt-4.12025.10	90.4	96.1	82.3	80.8	81.5	84.3	71.1	99.7
gpt-4.1-mini2025.10	89.9	95.7	81.5	79.3	81	90.2	76.3	99.8
gpt-4.1-mini2025.10	89.6	95.9	80.5	78.9	78	90.2	71.1	99.9
gpt-4o2025.10	88.8	95.3	79.4	77	80	80.4	73.7	99.5
gpt-4o2025.10	87.8	95.7	76.3	70.9	75.1	84.3	73.7	99.5
claude-opus-42025.10	86.8	89.3	83.3	83.6	82	84.3	81.6	99.9
gpt-4.1-nano2025.10	85.1	92.3	74.8	73.7	75.1	84.3	63.2	98.7
llama-4-maverick2025.10	84.5	91.1	75.1	74.2	74.1	78.4	63.2	97.6
claude-sonnet-42025.10	83.4	86.9	78.4	76.5	79.5	72.5	86.8	100
gpt-4o-mini2025.10	82.8	90.5	71.7	71.8	70.2	80.4	65.8	98.2
llama-3.1-405b-instruct2025.10	82.3	90.7	70.2	63.8	70.7	76.5	71.1	98.5
llama-3.1-405b-instruct2025.10	81.4	90	69.2	68.1	68.8	68.6	65.8	98.1
gpt-4.1-nano2025.10	80.8	88.7	69.4	65.7	70.2	70.6	71.1	98.2
gpt-4o2025.10	80.8	86.2	73	72.3	71.7	72.5	71.1	100
gpt-4o-mini2025.10	79.9	86.2	70.7	68.5	65.4	92.2	65.8	99.2
llama-4-maverick2025.10	79.7	86.2	70.4	66.2	74.1	68.6	63.2	97.7
llama-3.1-70b-instruct2025.10	79.5	88.4	66.8	62.4	66.3	68.6	60.5	97.4
llama-4-maverick2025.10	78.9	86.4	68.1	64.8	64.4	68.6	78.9	100
llama-4-scout2025.10	78.4	86.4	66.8	62	66.3	80.4	60.5	95.8
gpt-4.1-mini2025.10	78.3	85.7	67.6	62	66.8	82.4	65.8	100
llama-3.1-70b-instruct2025.10	77.8	85.2	67.4	63.8	67.3	72.5	68.4	97.3
llama-4-scout2025.10	76.7	83.9	66.3	60.1	63.4	74.5	71.1	95.8
gpt-4.12025.10	75.2	80.5	67.6	70	67.3	64.7	57.9	100
gpt-4.12025.10	75	79.8	68.1	70	67.8	62.7	60.5	100
llama-3.1-405b-instruct2025.10	71.1	76	64	62.4	65.9	64.7	63.2	100
llama-3.1-405b-instruct2025.10	70.1	77.1	60.2	56.8	60	60.8	63.2	99.8
llama-4-maverick2025.10	69	74.2	61.4	60.6	60.5	66.7	65.8	100
llama-4-scout2025.10	69	75.7	59.4	57.7	58	64.7	60.5	99.9
gpt-4.1-nano2025.10	69	75	60.4	55.4	57.1	74.5	63.2	100
gpt-4.1-mini2025.10	68.6	74.1	60.7	56.3	58	76.5	55.3	100
gpt-4o2025.10	67.7	72.8	60.4	59.2	59	62.7	42.1	98.8
llama-3.1-70b-instruct2025.10	62.2	67.6	54.5	51.2	55.6	54.9	60.5	100
llama-4-scout2025.10	62.2	68	54	50.7	51.2	54.9	73.7	99.7
llama-3.1-70b-instruct2025.10	60.8	66	53.2	52.6	51.7	43.1	65.8	100
gpt-4o-mini2025.10	59.1	63	53.5	53.1	48.8	47.1	57.9	100
gpt-4.1-nano2025.10	58.9	60.6	56.3	51.6	58	60.8	68.4	100
gpt-4o-mini2025.10	52.5	52.8	52.2	50.7	51.2	45.1	55.3	100
gemini-2.5-flash2025.10	51.6	56.4	44.7	38.5	42.4	58.8	57.9	57.8
gemini-2.5-flash2025.10	49.7	54.2	43.2	36.2	43.9	58.8	55.3	52.5
gemini-2.5-flash2025.10	44	47.8	38.6	30	43.9	49	44.7	47
gemini-2.5-flash2025.10	23.1	29.3	14.1	8	11.7	29.4	23.7	30.6