# Annotator Modeling on AITA situation Source: [Annotator Modeling on AITA situation]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:31:09.831Z Evaluation of Annotator Modeling performance on the AITA situation dataset, typically measured using Accuracy and F1 score. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Annotator Modeling | | Dataset (including any stated split) | AITA situation | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 19 of 19 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. - [F1](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | Both Categorizations | 68.1 | Not recorded | 6944dbe0423ea422657298ac | | Relationships | 67.8 | Not recorded | 6944dbe0423ea422657298ac | | C3: Parenting & Discipline | 67.7 | Not recorded | 6944dbe0423ea422657298ac | | C8: Family Disputes | 67.7 | Not recorded | 6944dbe0423ea422657298ac | | Theory Based Categorization | 67.6 | Not recorded | 6944dbe0423ea422657298ac | | Demographics | 67.4 | Not recorded | 6944dbe0423ea422657298ac | | C1: Financial Issues | 67.3 | Not recorded | 6944dbe0423ea422657298ac | | C2: Acc. Social Needs | 67.3 | Not recorded | 6944dbe0423ea422657298ac | | No Comments | 67 | Not recorded | 6944dbe0423ea422657298ac | | C4: Family Struct. Change | 66.9 | Not recorded | 6944dbe0423ea422657298ac | | C6: Rel. Issues w/Women | 66.8 | Not recorded | 6944dbe0423ea422657298ac | | C9: Negative Affect | 66.8 | Not recorded | 6944dbe0423ea422657298ac | | Experiences | 66.7 | Not recorded | 6944dbe0423ea422657298ac | | C7: Sympathy & Support | 66.5 | Not recorded | 6944dbe0423ea422657298ac | | Automatic Categorization | 66.3 | Not recorded | 6944dbe0423ea422657298ac | | Attitudes | 66.1 | Not recorded | 6944dbe0423ea422657298ac | | All Comments | 65.9 | Not recorded | 6944dbe0423ea422657298ac | | C10: Food & Meals | 65.6 | Not recorded | 6944dbe0423ea422657298ac | | C5: Rel. Issues w/Men | 64.8 | Not recorded | 6944dbe0423ea422657298ac | ### Both Categorizations Accuracy: 68.1. Result ID: 6996bb635698665ec4062d7e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Diverse Sampling"}. This record does not assert independent reproduction. ### Relationships Accuracy: 67.8. Result ID: 6996bb635698665ec4062d73. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Theory Based High-level Categorization","annotator\_percentage\_with\_5+\_posts":56}. This record does not assert independent reproduction. ### C3: Parenting & Discipline Accuracy: 67.7. Result ID: 6996bb635698665ec4062d76. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Automatic High-level Categorization","annotator\_percentage\_with\_5+\_posts":33}. This record does not assert independent reproduction. ### C8: Family Disputes Accuracy: 67.7. Result ID: 6996bb635698665ec4062d7b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Automatic High-level Categorization","annotator\_percentage\_with\_5+\_posts":32}. This record does not assert independent reproduction. ### Theory Based Categorization Accuracy: 67.6. Result ID: 6996bb635698665ec4062d7f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Diverse Sampling"}. This record does not assert independent reproduction. ### Demographics Accuracy: 67.4. Result ID: 6996bb635698665ec4062d71. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Theory Based High-level Categorization","annotator\_percentage\_with\_5+\_posts":69}. This record does not assert independent reproduction. ### C1: Financial Issues Accuracy: 67.3. Result ID: 6996bb635698665ec4062d74. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Automatic High-level Categorization","annotator\_percentage\_with\_5+\_posts":30}. This record does not assert independent reproduction. ### C2: Acc. Social Needs Accuracy: 67.3. Result ID: 6996bb635698665ec4062d75. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Automatic High-level Categorization","annotator\_percentage\_with\_5+\_posts":42}. This record does not assert independent reproduction. ### No Comments Accuracy: 67. Result ID: 6996bb635698665ec4062d6e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Baseline Tests","annotator\_percentage\_with\_5+\_posts":0}. This record does not assert independent reproduction. ### C4: Family Struct. Change Accuracy: 66.9. Result ID: 6996bb635698665ec4062d77. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Automatic High-level Categorization","annotator\_percentage\_with\_5+\_posts":33}. This record does not assert independent reproduction. ### C6: Rel. Issues w/Women Accuracy: 66.8. Result ID: 6996bb635698665ec4062d79. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Automatic High-level Categorization","annotator\_percentage\_with\_5+\_posts":42}. This record does not assert independent reproduction. ### C9: Negative Affect Accuracy: 66.8. Result ID: 6996bb635698665ec4062d7c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Automatic High-level Categorization","annotator\_percentage\_with\_5+\_posts":39}. This record does not assert independent reproduction. ### Experiences Accuracy: 66.7. Result ID: 6996bb635698665ec4062d72. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Theory Based High-level Categorization","annotator\_percentage\_with\_5+\_posts":48}. This record does not assert independent reproduction. ### C7: Sympathy & Support Accuracy: 66.5. Result ID: 6996bb635698665ec4062d7a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Automatic High-level Categorization","annotator\_percentage\_with\_5+\_posts":24}. This record does not assert independent reproduction. ### Automatic Categorization Accuracy: 66.3. Result ID: 6996bb635698665ec4062d80. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Diverse Sampling"}. This record does not assert independent reproduction. ### Attitudes Accuracy: 66.1. Result ID: 6996bb635698665ec4062d70. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Theory Based High-level Categorization","annotator\_percentage\_with\_5+\_posts":68}. This record does not assert independent reproduction. ### All Comments Accuracy: 65.9. Result ID: 6996bb635698665ec4062d6f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Baseline Tests","annotator\_percentage\_with\_5+\_posts":100}. This record does not assert independent reproduction. ### C10: Food & Meals Accuracy: 65.6. Result ID: 6996bb635698665ec4062d7d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Automatic High-level Categorization","annotator\_percentage\_with\_5+\_posts":23}. This record does not assert independent reproduction. ### C5: Rel. Issues w/Men Accuracy: 64.8. Result ID: 6996bb635698665ec4062d78. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"sampling\_strategy":"Automatic High-level Categorization","annotator\_percentage\_with\_5+\_posts":37}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.