Disagreement Detection on CEFR-SP (Rater B)
0.539ROC-AUClogprob label
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| logprob labelLLM Judge=Qwen3, Rater=B2026.05 | 0.539 | 0.586 | 85 | |
| ModernBERT most2LLM Judge=Qwen3, Rater=B, Embedding Model=ModernBERT [6], Reasoning Chain=true2026.05 | 0.537 | 0.61 | 85 | |
| LaBSELLM Judge=Qwen3, Rater=B, Embedding Model=LaBSE, Method Variant=most22026.05 | 0.533 | 0.584 | 75 | |
| multilingual-e5-large-instructLLM Judge=Qwen3, Rater=B, Embedding Model=multilingual-e5-large-instruct, Method Variant=most22026.05 | 0.526 | 0.592 | 75 | |
| deberta-v3-largeLLM Judge=Qwen3, Rater=B, Embedding Model=deberta-v3-large, Method Variant=most22026.05 | 0.523 | 0.585 | 75 | |
| e5-large-v2LLM Judge=Qwen3, Rater=B, Embedding Model=e5-large-v2, Method Variant=most22026.05 | 0.52 | 0.581 | 70 | |
| bge-large-en-v1.5LLM Judge=Qwen3, Rater=B, Embedding Model=bge-large-en-v1.5, Method Variant=most22026.05 | 0.506 | 0.577 | 70 | |
| gte-large-en-v1.5LLM Judge=Qwen3, Rater=B, Embedding Model=gte-large-en-v1.5, Method Variant=most22026.05 | 0.505 | 0.574 | 75 | |
| sentence-t5-largeLLM Judge=Qwen3, Rater=B, Embedding Model=sentence-t5-large, Method Variant=most22026.05 | 0.503 | 0.569 | 75 | |
| all-roberta-large-v1LLM Judge=Qwen3, Rater=B, Embedding Model=all-roberta-large-v1, Method Variant=most22026.05 | 0.503 | 0.57 | 70 | |
| instructor-largeLLM Judge=Qwen3, Rater=B, Embedding Model=instructor-large, Method Variant=most22026.05 | 0.503 | 0.573 | 80 | |
| p i exactwLLM Judge=Qwen3, Rater=B, Reasoning Chain=true2026.05 | 0.501 | 0.558 | 75 | |
| sentence-t5-baseLLM Judge=Qwen3, Rater=B, Embedding Model=sentence-t5-base, Method Variant=most22026.05 | 0.496 | 0.567 | 80 | |
| ModernBERT exactwLLM Judge=Qwen3, Rater=B, Embedding Model=ModernBERT [6], Reasoning Chain=true2026.05 | 0.424 | 0.508 | 30 |