LLM Evaluation Performance on FairEval
0.6375AccuracyCHATEVAL (GPT)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| CHATEVAL (GPT)Evaluator Role Strategy=One Evaluator, Many Roles, Conversation Mode=Cross-Role Agent Conversations, LLM Queries=880 (80×11)2026.02 | 0.6375 | 0.4587 | 0.3766 | 0.3584 | |
| MILE-RefHumEval (DeepSeek+Mistral+GPT+Llama+Gemma+Phi+Qwen)Evaluator Role Strategy=One Role, Many Evaluators, Conversation Mode=No Conversations, LLM Queries=560 (80×7)2026.02 | 0.6375 | 0.4987 | 0.3904 | 0.3742 | |
| CHATEVAL (Qwen)Evaluator Role Strategy=One Evaluator, Many Roles, Conversation Mode=Cross-Role Agent Conversations2026.02 | 0.625 | 0.4779 | 0.3462 | 0.3305 | |
| MILE-RefHumEval-Conv (GPT→Mistral→Qwen→Llama→Gemma→Phi→DeepSeek)Evaluator Role Strategy=One Role, Many Evaluators, Conversation Mode=With Conversations2026.02 | 0.6125 | 0.4367 | 0.3251 | 0.308 | |
| MILE-RefHumEval-Conv (DeepSeek→Mistral→GPT→Llama→Gemma→Phi→Qwen)Evaluator Role Strategy=One Role, Many Evaluators, Conversation Mode=With Conversations2026.02 | 0.5875 | 0.4284 | 0.3266 | 0.3038 | |
| MILE-RefHumEval-Conv (Qwen→Mistral→GPT→Llama→Gemma→Phi→DeepSeek)Evaluator Role Strategy=One Role, Many Evaluators, Conversation Mode=With Conversations2026.02 | 0.5875 | 0.4249 | 0.2882 | 0.2797 | |
| MILE-RefHumEval-Conv (Phi→Mistral→GPT→Llama→Gemma→Qwen→DeepSeek)Evaluator Role Strategy=One Role, Many Evaluators, Conversation Mode=With Conversations, LLM Queries=1760 (80×22)2026.02 | 0.5875 | 0.4242 | 0.3005 | 0.2857 | |
| CHATEVAL (Gemma)Evaluator Role Strategy=One Evaluator, Many Roles, Conversation Mode=Cross-Role Agent Conversations2026.02 | 0.575 | 0.4163 | 0.2778 | 0.2663 | |
| MILE-RefHumEval-Conv (Gemma→DeepSeek→GPT→Llama→Qwen→Phi→Mistral)Evaluator Role Strategy=One Role, Many Evaluators, Conversation Mode=With Conversations2026.02 | 0.575 | 0.41 | 0.2501 | 0.2366 | |
| CHATEVAL (Phi)Evaluator Role Strategy=One Evaluator, Many Roles, Conversation Mode=Cross-Role Agent Conversations2026.02 | 0.55 | 0.4388 | 0.239 | 0.2332 | |
| MILE-RefHumEval-Conv (Llama→Mistral→GPT→Phi→Gemma→Qwen→DeepSeek)Evaluator Role Strategy=One Role, Many Evaluators, Conversation Mode=With Conversations2026.02 | 0.5375 | 0.3845 | 0.1979 | 0.1886 | |
| CHATEVAL (Deepseek)Evaluator Role Strategy=One Evaluator, Many Roles, Conversation Mode=Cross-Role Agent Conversations2026.02 | 0.525 | 0.3916 | 0.2496 | 0.2278 | |
| CHATEVAL (Llama)Evaluator Role Strategy=One Evaluator, Many Roles, Conversation Mode=Cross-Role Agent Conversations2026.02 | 0.525 | 0.3905 | 0.3106 | 0.2551 | |
| CHATEVAL (Mistral)Evaluator Role Strategy=One Evaluator, Many Roles, Conversation Mode=Cross-Role Agent Conversations2026.02 | 0.45 | 0.3326 | 0.1477 | 0.1248 |