Question Answering on Riddle (test)
67.84AccuracyOracle
Evaluation results
| Method | Links | |
|---|---|---|
| OracleModel Size=Large, Noise Type=Uniform2026.04 | 67.84 | |
| REALMModel Size=Large, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 65.45 | |
| REALMModel Size=Large, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 62.31 | |
| NOISYModel Size=Large, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 61.49 | |
| REALMModel Size=Large, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 60.43 | |
| NOISYModel Size=Large, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 57.06 | |
| OracleModel Size=Base, Noise Type=Uniform2026.04 | 54.12 | |
| NOISYModel Size=Large, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 50.35 | |
| REALMModel Size=Base, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 49.61 | |
| REALMModel Size=Base, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 46.63 | |
| REALMModel Size=Base, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 46.04 | |
| NOISYModel Size=Base, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 45.84 | |
| NOISYModel Size=Base, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 42 | |
| OracleModel Size=Small2026.04 | 39.02 | |
| OracleModel Size=Small, Noise Type=Uniform2026.04 | 39.02 | |
| NOISYModel Size=Base, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 36.2 | |
| REALMModel Size=Small, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 34.71 | |
| REALMModel Size=Small, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 34.71 | |
| REALMModel Size=Small, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 32.16 | |
| REALMModel Size=Small, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 31.49 | |
| noisyModel Size=Small, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 31.25 | |
| NOISYModel Size=Small, Expertise Distribution=Dist. 2, Noise Type=Uniform2026.04 | 31.25 | |
| NOISYModel Size=Small, Expertise Distribution=Dist. 3, Noise Type=Uniform2026.04 | 29.69 | |
| NOISYModel Size=Small, Expertise Distribution=Dist. 1, Noise Type=Uniform2026.04 | 23.8 | |
| OracleModel Size=Large, Expertise Distribution=Dist. 32026.04 | 0.6784 | |
| REALMModel Size=Large, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 0.6231 | |
| REALMModel Size=Large, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 0.6133 | |
| REALMModel Size=Large, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 0.6125 | |
| NOISYModel Size=Large, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 0.5792 | |
| NOISYModel Size=Large, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 0.5706 | |
| NOISYModel Size=Large, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 0.5616 | |
| OracleModel Size=Base, Expertise Distribution=Dist. 32026.04 | 0.5412 | |
| REALMModel Size=Base, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 0.4694 | |
| REALMModel Size=Base, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 0.4663 | |
| REALMModel Size=Base, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 0.4647 | |
| NOISYModel Size=Base, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 0.4247 | |
| NOISYModel Size=Base, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 0.4208 | |
| NOISYModel Size=Base, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 0.42 | |
| OracleModel Size=Small, Expertise Distribution=Dist. 32026.04 | 0.3902 | |
| REALMModel Size=Small, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 0.3259 | |
| REALMModel Size=Small, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 0.3227 | |
| REALMModel Size=Small, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 0.3216 | |
| NOISYModel Size=Small, Noise Type=Uniform, Expertise Distribution=Dist. 32026.04 | 0.2969 | |
| NOISYModel Size=Small, Noise Type=Asymmetric, Expertise Distribution=Dist. 32026.04 | 0.291 | |
| NOISYModel Size=Small, Noise Type=Systematic, Expertise Distribution=Dist. 32026.04 | 0.2816 |