LLM Judge Reliability Assessment on Therapeutic Dialogue Evaluation Set

0.989ICC (C,1)TheraJudge (gen 16)

Evaluation results

Method
TheraJudge (gen 16)2026.06	0.989	0.948	0.981	0.05	—	1
TheraJudge (gen 16)2026.06	0.983	0.918	0.977	0.077	—	1
TheraJudge (gen 16)2026.06	0.96	0.9	0.929	0.085	—	1
TheraJudge (gen 16)2026.06	0.96	0.882	0.946	0.106	—	1
TheraJudge (gen 16)2026.06	0.949	—	0.933	0.254	7	1
o4-mini2026.06	0.948	0.744	0.786	0.233	—	2
TheraJudge (gen 16)2026.06	0.945	0.694	0.934	0.291	—	1
TheraJudge (gen 16)2026.06	0.932	0.709	0.919	0.27	—	1
Qwen-2.5-7B-SFT2026.06	0.929	0.852	0.911	0.114	—	3
Qwen-2.5-7B-SFT2026.06	0.921	0.82	0.926	0.16	—	2
o4-mini2026.06	0.918	0.638	0.908	0.34	—	2
Claude-3.7-Sonnet2026.06	0.915	0.83	0.915	0.142	—	3
Claude-3.7-Sonnet2026.06	0.906	0.429	0.474	0.528	—	2
Claude-3.7-Sonnet2026.06	0.9	0.734	0.742	0.258	—	3
Qwen-2.5-7B-SFT2026.06	0.895	0.805	0.886	0.178	—	4
o4-mini2026.06	0.883	0.476	0.499	0.469	—	3
Claude-3.7-Sonnet2026.06	0.881	0.764	0.837	0.216	—	4
TheraJudge (gen 16)2026.06	0.879	0.561	0.848	0.901	—	1
Gemini-2.5-Flash2026.06	0.878	0.522	0.877	0.439	—	5
Qwen-2.5-7B-SFT2026.06	0.874	0.459	0.866	0.518	—	4
o4-mini2026.06	0.871	0.578	0.66	0.356	—	4
o4-mini2026.06	0.871	0.636	0.592	0.302	—	2
Qwen-2.5-7B-ZS2026.06	0.87	0.471	0.712	0.453	—	3
GPT-4o2026.06	0.856	0.655	0.681	0.31	—	6
Gemini-2.5-Flash2026.06	0.855	0.557	0.682	0.398	—	5
GPT-4o2026.06	0.849	0.65	0.475	0.324	—	6
Gemini-2.5-Flash2026.06	0.838	0.401	0.38	0.517	—	5
GPT-4o2026.06	0.835	0.331	0.288	0.56	—	6
Claude-3.7-Sonnet2026.06	0.83	—	0.731	0.394	5	3
Qwen-2.5-7B-SFT2026.06	0.827	—	0.783	0.229	6	2
GPT-4o2026.06	0.823	0.549	0.485	0.334	—	4
Qwen-2.5-7B-SFT2026.06	0.803	0.646	0.743	0.216	—	5
Qwen-2.5-7B-ZS2026.06	0.802	0.573	0.669	0.376	—	7
GPT-4o2026.06	0.8	0.407	0.457	0.517	—	5
Claude-3.7-Sonnet2026.06	0.791	0.563	0.806	0.394	—	6
Qwen-2.5-7B-ZS2026.06	0.761	0.394	0.74	0.493	—	6
Qwen-2.5-7B-SFT2026.06	0.76	0.636	0.695	0.175	—	2
GPT-4o2026.06	0.739	—	0.415	0.478	5	5
Gemini-2.5-Flash2026.06	0.734	0.271	0.385	0.561	—	7
Claude-3.7-Sonnet2026.06	0.73	0.394	0.743	0.594	—	3
o4-mini2026.06	0.727	—	0.529	0.418	5	4
Claude-3.7-Sonnet2026.06	0.685	0.333	0.597	0.628	—	2
Qwen-2.5-7B-ZS2026.06	0.65	0.45	0.499	0.374	—	7
Qwen-2.5-7B-ZS2026.06	0.623	—	0.509	0.423	3	6
Gemini-2.5-Flash2026.06	0.621	—	0.409	0.586	3	7
Qwen-2.5-7B-ZS2026.06	0.616	0.164	0.602	0.544	—	7
Qwen-2.5-7B-SFT2026.06	0.605	0.429	0.451	0.242	—	3
GPT-4o2026.06	0.532	0.267	0.243	0.559	—	4
Qwen-2.5-7B-ZS2026.06	0.519	0.31	0.263	0.292	—	5
GPT-4o2026.06	0.48	0.116	0.279	0.741	—	4
Gemini-2.5-Flash2026.06	0.377	0.047	0.222	0.79	—	5
Gemini-2.5-Flash2026.06	0.362	0.137	0.18	0.644	—	7
o4-mini2026.06	0.342	0.069	0.14	0.605	—	6
Gemini-2.5-Flash2026.06	0.306	0.011	0.137	0.755	—	7
o4-mini2026.06	0.259	0.081	0.117	0.621	—	6
Qwen-2.5-7B-ZS2026.06	0.145	0.155	0.079	0.427	—	7