Social Dimension Classification on P1SCO Perspectivist N=1104 (test)
47.1Mac-F1Mistral-7B-v0.3
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Mistral-7B-v0.3Setting=Zero-shot2026.06 | 47.1 | 52.8 | 51.8 | |
| Mistral-7B-v0.3Setting=Few-shot2026.06 | 46 | 51.5 | 52.9 | |
| RandomSetting=Baselines2026.06 | 42.8 | 51.4 | 43.4 | |
| RoBERTa+demo (Perspectivist)Setting=Fine-tuned2026.06 | 42.2 | 50.5 | 41.4 | |
| RoBERTa (Majority)Setting=Fine-tuned2026.06 | 41.4 | 49.1 | 71.1 | |
| RoBERTatext (Perspectivist)Setting=Fine-tuned2026.06 | 41 | 49.7 | 38.9 | |
| DeepSeek-R1-14BSetting=Few-shot2026.06 | 40.9 | 49.2 | 51.2 | |
| DeepSeek-R1-32BSetting=Few-shot2026.06 | 40.5 | 46 | 52.3 | |
| Llama-3.1-8B-ablSetting=Few-shot, Ablated=true2026.06 | 39.1 | 47.9 | 51.4 | |
| GPT-4o-miniSetting=Zero-shot2026.06 | 36.8 | 42 | 51.6 | |
| Llama-3.1-8B-ablSetting=Zero-shot, Ablated=true2026.06 | 36.4 | 41.4 | 50.8 | |
| DeepSeek-R1-14BSetting=Zero-shot2026.06 | 35.9 | 43.8 | 50.3 | |
| Qwen3-8BSetting=Few-shot2026.06 | 34.1 | 45.2 | 45.2 | |
| DeepSeek-R1-32BSetting=Zero-shot2026.06 | 33.3 | 38.3 | 50.6 | |
| Qwen3-8BSetting=Zero-shot2026.06 | 33.1 | 38.9 | 45.4 | |
| Majority-classSetting=Baselines2026.06 | 30.8 | 60.7 | 42.8 | |
| Qwen3-8B-ablSetting=Few-shot, Ablated=true2026.06 | 28.7 | 42.6 | 46.1 | |
| Llama-3.1-8BSetting=Few-shot2026.06 | 28.4 | 39.8 | 48.1 | |
| GPT-4o-miniSetting=Few-shot2026.06 | 27.1 | 31.6 | 49.7 | |
| DeepSeek-R1-7BSetting=Few-shot2026.06 | 26.5 | 32.2 | 46.4 | |
| Llama-3.1-8BSetting=Zero-shot2026.06 | 21.3 | 32.7 | 46 | |
| Qwen3-8B-ablSetting=Zero-shot, Ablated=true2026.06 | 16.8 | 28.3 | 44 | |
| Choi et al. (2020)Setting=Baselines2026.06 | 15.1 | 26.1 | 59.9 | |
| DeepSeek-R1-7BSetting=Zero-shot2026.06 | 6.6 | 7.2 | 43.5 |