Research Paper Reasoning and Comprehension on ArxivRollBench 2025a (val)

44.3Valid Accuracygoogle/gemini-2.5-flash

Evaluation results

Method
google/gemini-2.5-flash2025.07	44.3	44.3	100
anthropic/claude-3.5-sonnet2025.07	43.1	43.1	100
anthropic/claude-3.7-sonnet2025.07	42.9	42.9	100
anthropic/claude-opus-42025.07	42.7	42.7	100
anthropic/claude-sonnet-42025.07	41.2	41.2	100
deepseek/deepseek-chat-v3-03242025.07	40.5	40.5	100
google/gemini-2.0-flash-0012025.07	39.5	39.5	100
moonshotai/kimi-k22025.07	37	37	100
openai/gpt-4o2025.07	36.2	36.2	100
openai/gpt-42025.07	35.3	35.3	100
nvidia/Llama-3.1-Nemotron-70B-Instruct-HF2025.07	33	33	100
meta-llama/Llama-3.1-70B-Instruct2025.07	32.2	32.2	100
meta-llama/Llama-3.3-70B-Instruct2025.07	30.1	30.1	100
Qwen/Qwen2.5-7B-Instruct2025.07	29.6	29.6	100
Qwen/Qwen2-7B-Instruct2025.07	29	29	100
Qwen/Qwen3-8B2025.07	28.4	28.4	100
deepseek/deepseek-chat-v3.12025.07	28.4	28.4	100
Qwen/Qwen2.5-7B2025.07	28	28	100
meta-llama/Llama-3.1-8B-Instruct2025.07	27.5	27.5	100
meta-llama/Llama-3.2-3B2025.07	23.4	23.4	100
meta-llama/Meta-Llama-3-8B2025.07	22.6	22.6	100
microsoft/phi-22025.07	22.3	22.3	100
meta-llama/Llama-3.1-8B2025.07	21.8	21.8	100
meta-llama/Llama-3.2-1B2025.07	21.7	21.7	100
Qwen/Qwen2.5-Math-7B2025.07	21.4	21.4	100
microsoft/Phi-4-reasoning-plus2025.07	18.3	18.3	100
microsoft/Phi-3.5-mini-instruct2025.07	16.9	16.9	100
Qwen/Qwen2.5-72B-Instruct2025.07	13.4	13.4	100
microsoft/Phi-3-mini-4k-instruct2025.07	12.8	12.8	100
microsoft/phi-1_52025.07	12.5	12.5	100
Qwen/Qwen3-32B2025.07	9.2	9.2	100
meta-llama/Llama-2-7b-chat-hf2025.07	8.4	8.4	100
Qwen/Qwen2.5-Math-7B-Instruct2025.07	7.3	7.3	100
Qwen/Qwen3-14B2025.07	7.2	7.2	100
microsoft/Phi-4-reasoning2025.07	4.8	4.8	100
microsoft/phi-12025.07	4.4	4.4	100
google/gemini-2.5-pro2025.07	0.2	0.2	100
deepseek/deepseek-r1-05282025.07	0	0	100