Question Answering Difficulty Prediction on QA Difficulty (20% test)
0.65MAE (Mean)CrowdLLM
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CrowdLLMBackbone=Gemma 3-12B2025.12 | 0.65 | 0.74 | 0.63 | 1.49 | 0.89 | 0.83 | 1.06 | 1.18 | 2.08 | 1.43 | 0.71 | 0.74 | |
| LLMProtocol=Self-Consistency (SC), Backbone=Gemma 3-12B2025.12 | 0.68 | 0.79 | 1.02 | 0.98 | 1.03 | 0.87 | 1.02 | 1.22 | 1.23 | 1.25 | 0.36 | 1.01 | |
| LLMProtocol=zero-shot, Backbone=Gemma 3-12B2025.12 | 0.71 | 0.81 | 1.04 | 1.02 | 1.05 | 0.9 | 1.04 | 1.24 | 1.28 | 1.26 | 0.33 | 1.06 | |
| LLMProtocol=multi-persona, Backbone=Gemma 3-12B2025.12 | 0.73 | 0.82 | 1 | 1.06 | 1.02 | 1.02 | 1.16 | 1.38 | 1.46 | 1.41 | 0.48 | 0.93 | |
| VAE2025.12 | 0.76 | 0.9 | 0.71 | 1.41 | 1.11 | 0.98 | 1.2 | 1.14 | 1.85 | 1.53 | 0.68 | 0.88 | |
| Random2025.12 | 1.21 | 1.41 | 1.52 | 1.98 | 1.81 | 1.43 | 1.73 | 2.02 | 2.38 | 2.26 | 0.49 | 1.29 |