Evaluator Reliability Assessment on GlobalHealthAtlas 100 human-annotated scores 1.0 (test)
1.4216MAEqwen-plus
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| qwen-plusprompting=identical generic prompt2026.01 | 1.4216 | 4.255 | 2.0001 | 0.9586 | 34.5 | 0.3825 | 4.17 | |
| Public-Evaluator2026.01 | 1.4259 | 4.1296 | 1.9558 | 0.9735 | 55.33 | 0.2772 | 3.67 | |
| gpt-5-miniprompting=identical generic prompt2026.01 | 1.4383 | 4.1617 | 1.9431 | 0.9733 | 30.67 | 0.359 | 2 | |
| qwen3-32bprompting=identical generic prompt2026.01 | 1.4811 | 4.519 | 2.0768 | 0.9519 | 14.56 | 0.6827 | 15.44 | |
| qwen3-maxprompting=identical generic prompt2026.01 | 1.4833 | 4.2767 | 2.0104 | 0.9593 | 35.88 | 0.3182 | 1.53 | |
| claude-3-5-sonnet-20241022prompting=identical generic prompt2026.01 | 1.5233 | 4.65 | 2.1234 | 0.9186 | 37.67 | 0.318 | 2 | |
| glm-4.7prompting=identical generic prompt2026.01 | 1.575 | 4.805 | 2.1488 | 0.9512 | 46.33 | 0.3226 | 3.5 | |
| deepseek-v3.2prompting=identical generic prompt2026.01 | 1.5883 | 4.8283 | 2.1459 | 0.9515 | 12.5 | 0.7674 | 20.5 | |
| qwen3-8bprompting=identical generic prompt2026.01 | 1.734 | 5.9124 | 2.3722 | 0.9133 | 32.64 | 0.4224 | 4.51 | |
| Clinical-Judgeprompting=few-shot prompting2026.01 | 2.0122 | 7.1094 | 2.6377 | 0.3591 | 48.67 | 0.481 | 10.67 | |
| gemini-3-flash-previewprompting=identical generic prompt2026.01 | 2.0375 | 7.2687 | 2.6232 | 0.897 | 35.33 | 0.5837 | 17.5 | |
| Expert-Guided Evaluatorprompting=structured reasoning2026.01 | 2.1633 | 9.7833 | 3.084 | 0.8541 | 42.33 | 0.4677 | 7.5 | |
| qwen-flashprompting=identical generic prompt2026.01 | 2.4033 | 10.48 | 3.1473 | 0.8744 | 36.67 | 0.7461 | 27.44 |