Distractor Effectiveness on EduAgent (test)
0.7933Agreement AccuracyQG-SMS
Evaluation results
| Method | Links | ||
|---|---|---|---|
| QG-SMSScoring Protocol=Pairwise LLM-based, Base Model=GPT-4o2025.03 | 0.7933 | 74.67 | |
| KDA_largeScoring Protocol=Individual Scoring2025.03 | 0.7733 | — | |
| VanillaScoring Protocol=Pairwise LLM-based, Base Model=GPT-4o2025.03 | 0.7333 | 64 | |
| MetricsScoring Protocol=Pairwise LLM-based, Base Model=GPT-4o2025.03 | 0.72 | 62.67 | |
| ReferenceScoring Protocol=Pairwise LLM-based, Base Model=GPT-4o2025.03 | 0.6933 | 56 | |
| ChatEvalScoring Protocol=Pairwise LLM-based, Base Model=GPT-4o2025.03 | 0.6933 | 56 | |
| QSalienceScoring Protocol=Individual Scoring2025.03 | 0.68 | — | |
| SwapScoring Protocol=Pairwise LLM-based, Base Model=GPT-4o2025.03 | 0.68 | 53.33 | |
| BERTScoreScoring Protocol=Individual Scoring2025.03 | 0.6533 | — | |
| CoTScoring Protocol=Pairwise LLM-based, Base Model=GPT-4o2025.03 | 0.6 | 28 |