Annotator Modeling on AITA situation
68.1AccuracyBoth Categorizations
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Both Categorizationssampling_strategy=Diverse Sampling2025.12 | 68.1 | 56 | |
| Relationshipssampling_strategy=Theory Based High-level Categorization, annotator_percentage_with_5+_posts=562025.12 | 67.8 | 56 | |
| C3: Parenting & Disciplinesampling_strategy=Automatic High-level Categorization, annotator_percentage_with_5+_posts=332025.12 | 67.7 | 57.2 | |
| C8: Family Disputessampling_strategy=Automatic High-level Categorization, annotator_percentage_with_5+_posts=322025.12 | 67.7 | 56.6 | |
| Theory Based Categorizationsampling_strategy=Diverse Sampling2025.12 | 67.6 | 56.7 | |
| Demographicssampling_strategy=Theory Based High-level Categorization, annotator_percentage_with_5+_posts=692025.12 | 67.4 | 58.6 | |
| C1: Financial Issuessampling_strategy=Automatic High-level Categorization, annotator_percentage_with_5+_posts=302025.12 | 67.3 | 56.2 | |
| C2: Acc. Social Needssampling_strategy=Automatic High-level Categorization, annotator_percentage_with_5+_posts=422025.12 | 67.3 | 56.1 | |
| No Commentssampling_strategy=Baseline Tests, annotator_percentage_with_5+_posts=02025.12 | 67 | 56 | |
| C4: Family Struct. Changesampling_strategy=Automatic High-level Categorization, annotator_percentage_with_5+_posts=332025.12 | 66.9 | 56.6 | |
| C6: Rel. Issues w/Womensampling_strategy=Automatic High-level Categorization, annotator_percentage_with_5+_posts=422025.12 | 66.8 | 55.8 | |
| C9: Negative Affectsampling_strategy=Automatic High-level Categorization, annotator_percentage_with_5+_posts=392025.12 | 66.8 | 57.3 | |
| Experiencessampling_strategy=Theory Based High-level Categorization, annotator_percentage_with_5+_posts=482025.12 | 66.7 | 56.1 | |
| C7: Sympathy & Supportsampling_strategy=Automatic High-level Categorization, annotator_percentage_with_5+_posts=242025.12 | 66.5 | 57.9 | |
| Automatic Categorizationsampling_strategy=Diverse Sampling2025.12 | 66.3 | 55.6 | |
| Attitudessampling_strategy=Theory Based High-level Categorization, annotator_percentage_with_5+_posts=682025.12 | 66.1 | 57.4 | |
| All Commentssampling_strategy=Baseline Tests, annotator_percentage_with_5+_posts=1002025.12 | 65.9 | 55.9 | |
| C10: Food & Mealssampling_strategy=Automatic High-level Categorization, annotator_percentage_with_5+_posts=232025.12 | 65.6 | 56.3 | |
| C5: Rel. Issues w/Mensampling_strategy=Automatic High-level Categorization, annotator_percentage_with_5+_posts=372025.12 | 64.8 | 55.8 |