Social Dimension Classification on P1SCO Majority N=1104 (test)
0.425Mac-F1RoBERTa+demo (Perspectivist)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| RoBERTa+demo (Perspectivist)Setting=Fine-tuned2026.06 | 0.425 | 0.552 | 0.43 | |
| RoBERTa (Majority)Setting=Fine-tuned2026.06 | 0.42 | 0.564 | 0.438 | |
| RoBERTatext (Perspectivist)Setting=Fine-tuned2026.06 | 0.405 | 0.531 | 0.416 | |
| GPT-4o-miniSetting=Zero-shot2026.06 | 0.359 | 0.454 | 0.202 | |
| Llama-3.1-8B-ablSetting=Few-shot, Ablated=true2026.06 | 0.338 | 0.493 | 0.2 | |
| DeepSeek-R1-14BSetting=Few-shot2026.06 | 0.334 | 0.451 | 0.196 | |
| Mistral-7B-v0.3Setting=Few-shot2026.06 | 0.332 | 0.447 | 0.206 | |
| DeepSeek-R1-32BSetting=Few-shot2026.06 | 0.329 | 0.446 | 0.211 | |
| GPT-4o-miniSetting=Few-shot2026.06 | 0.306 | 0.428 | 0.199 | |
| Mistral-7B-v0.3Setting=Zero-shot2026.06 | 0.305 | 0.403 | 0.183 | |
| DeepSeek-R1-32BSetting=Zero-shot2026.06 | 0.3 | 0.418 | 0.2 | |
| Llama-3.1-8B-ablSetting=Zero-shot, Ablated=true2026.06 | 0.296 | 0.417 | 0.194 | |
| Llama-3.1-8BSetting=Few-shot2026.06 | 0.291 | 0.446 | 0.176 | |
| Qwen3-8BSetting=Few-shot2026.06 | 0.275 | 0.385 | 0.131 | |
| DeepSeek-R1-14BSetting=Zero-shot2026.06 | 0.272 | 0.395 | 0.176 | |
| Qwen3-8BSetting=Zero-shot2026.06 | 0.259 | 0.327 | 0.133 | |
| Qwen3-8B-ablSetting=Few-shot, Ablated=true2026.06 | 0.249 | 0.372 | 0.153 | |
| DeepSeek-R1-7BSetting=Few-shot2026.06 | 0.225 | 0.331 | 0.146 | |
| Qwen3-8B-ablSetting=Zero-shot, Ablated=true2026.06 | 0.177 | 0.325 | 0.121 | |
| Llama-3.1-8BSetting=Zero-shot2026.06 | 0.151 | 0.269 | 0.131 | |
| Choi et al. (2020)Setting=Baselines2026.06 | 0.132 | 0.334 | 0.282 | |
| RandomSetting=Baselines2026.06 | 0.102 | 0.191 | 0.107 | |
| DeepSeek-R1-7BSetting=Zero-shot2026.06 | 0.062 | 0.112 | 0.111 | |
| Majority-classSetting=Baselines2026.06 | 0 | 0 | 0.103 |