Macrostructure Element Annotation on 64 narratives CHAT format
0.872Cohen's KappaHuman raters
Evaluation results
| Method | Links | |
|---|---|---|
| Human ratersAgreement type=Inter-human2026.05 | 0.872 | |
| DeepSeek-R1Model Type=API-based, Model Scale=Large-scale, Reasoning capability=Reasoning model2026.05 | 0.794 | |
| DeepSeek-V3Model Type=API-based, Model Scale=Large-scale, Reasoning capability=Non-reasoning counterpart2026.05 | 0.751 | |
| Qwen3-max-previewModel Type=API-based, Model Scale=Large-scale2026.05 | 0.725 | |
| DeepSeek-R1-Distill-Qwen-14BModel Type=Locally deployable, Model Scale=Lightweight, Backbone=Qwen-14B2026.05 | 0.53 |