Segment Intrusion on Segment Intrusion Restaurant domain 1.0 (test)

98SI-EHuman Performance

Evaluation results

Method
Human Performance2026.05	98	97	97	87	94.75
o4-mini2026.05	98	83.71	95	77.18	88.47
o3-mini2026.05	97.5	82.12	94	74.63	87.06
gemini-2.5-flash-preview-04-172026.05	97.5	82.5	76.27	63.98	80.06
claude-3.7-sonnet-202502192026.05	96.76	77.5	95	72.99	85.56
llama-3.3-70b-instruct-turbo2026.05	95.52	74.19	88.78	64.89	80.84
deepseek-v32026.05	95	78.5	85.57	70.22	82.32
gpt-4o2026.05	93.5	79.31	89.28	73.48	83.89
qwen2.5-72b-instruct-turbo2026.05	91.27	74	91.04	65.86	80.54
claude-3.5-haiku-202410222026.05	90.51	72.55	82	66.34	77.85
gemini-2.0-flash2026.05	90	66.5	86.28	63.77	76.64
gemini-2.0-flash-lite2026.05	79.82	62.57	76.24	66.67	71.32
qwen2.5-7b-instruct-turbo2026.05	74.81	53.5	40.1	31.75	50.04
llama-4-maverick-17b-128e-instruct-fp82026.05	55.94	45.59	21.48	19.82	35.71
llama-3.2-3b-instruct-turbo2026.05	27.07	20.76	10.49	7.5	16.45