Language Model Evaluation on ArxivRollBench 2026a

70.8Valid Accuracymoonshotai/kimi-k2.6

Evaluation Results

Method
moonshotai/kimi-k2.62025.07	70.8	5.6	7.9
gemini-3.5-flash2025.07	55.2	55.2	100
gpt-5.52025.07	54.4	54.4	100
claude-opus-4-72025.07	53.4	52.9	99
deepseek/deepseek-v4-pro2025.07	53.3	25.4	47.7
x-ai/grok-4.32025.07	52	51.8	99.7
claude-sonnet-4-62025.07	48.3	48.3	100
moonshotai/kimi-k2.52025.07	46	12.3	26.8
deepseek/deepseek-v4-flash2025.07	45.3	28	61.8
deepseek/deepseek-r1-05282025.07	43.2	14.3	33
claude-opus-4-62025.07	42.9	26.6	61.9
gpt-5.3-codex2025.07	42.1	40	95
claude-haiku-4-5-202510012025.07	40.8	40.2	98.7
gpt-5.42025.07	39.3	39.3	100
Qwen/Qwen3-14B2025.07	38.5	7.4	19.1
deepseek/deepseek-chat-v3-03242025.07	37.6	37.6	100
gpt-5.4-mini2025.07	35.5	35.4	99.8
x-ai/grok-4.202025.07	34.5	34.5	100
deepseek/deepseek-chat-v3.12025.07	33.9	33.9	100
deepseek/deepseek-v3.22025.07	31.8	31.8	100
Qwen/Qwen2.5-7B-Instruct2025.07	30	30	100
Qwen/Qwen2-7B-Instruct2025.07	29.5	29.4	99.8
Qwen/Qwen3-8B2025.07	29.3	29.3	99.8
Qwen/Qwen2.5-7B2025.07	28.2	28.2	100
meta-llama/Llama-3.1-8B-Instruct2025.07	27.7	27.7	100
Qwen/Qwen3-4B2025.07	27.4	19.7	71.8
mistralai/Mistral-7B-Instruct-v0.32025.07	26.2	26.2	100
mistralai/Mistral-7B-Instruct-v0.12025.07	25.8	25.8	100
tiiuae/Falcon3-10B-Instruct2025.07	23.8	23.8	99.9
mistralai/Mistral-7B-Instruct-v0.22025.07	23.7	22.7	95.6
meta-llama/Llama-3.2-3B2025.07	23.4	23.4	100
meta-llama/Llama-3.2-3B-Instruct2025.07	23.2	23.2	100
meta-llama/Llama-3.2-1B2025.07	22.1	22.1	100
meta-llama/Llama-3.1-8B2025.07	21.9	21.9	100
Qwen/Qwen2.5-Math-7B2025.07	21.6	21.2	98.2
microsoft/Phi-4-reasoning-plus2025.07	21.2	20.3	95.6
meta-llama/Llama-3.2-1B-Instruct2025.07	12.6	12.6	100
microsoft/Phi-4-reasoning2025.07	8.3	4.8	58.5
Qwen/Qwen2.5-Math-7B-Instruct2025.07	7.5	7.5	100
deepseek/deepseek-v3.2-speciale2025.07	0	0	0
gemini-3-flash-preview2025.07	0	0	100
gemini-3.1-pro-preview2025.07	0	0	100