Judge Agreement and Model Ranking on ComparIA benchmark leaderboard · SOTA2 Research

Judge Agreement and Model Ranking on ComparIA

0.156Kappa (κ)Gemma4-26B-A4B

Evaluation Results

Method	Links
Gemma4-26B-A4B2026.06		0.156	0.34	161.3	30.6	0.77	0.785	81
Gemma4-E4B2026.06		0.134	0.3	146.2	32.3	0.717	0.746	78
GPT-OSS-120B2026.06		0.132	0.37	102	30.9	0.74	0.769	70
GPT-OSS-20B2026.06		0.132	0.34	107.9	33	0.719	0.748	69
Llama-3.3-70B2026.06		0.127	0.32	68	35.3	0.751	0.767	48
Qwen3-32B2026.06		0.124	0.13	62.4	49.7	0.808	0.577	20
Qwen3.5-27B2026.06		0.102	0.14	82.9	47.8	0.845	0.66	42