# A Multi-Axis Annotation Scheme for Event Temporal Relations

Source: [A Multi-Axis Annotation Scheme for Event Temporal Relations](<https://www.sota2.com/research/paper/68fa64631e209131cea04422>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:03:11.991Z

Authors: Qiang Ning, Hao Wu, Dan Roth

Published: 2018-04-20T21:10:21.000Z

## Abstract

Existing temporal relation (TempRel) annotation schemes often have low inter-annotator agreements (IAA) even between experts, suggesting that the current annotation task needs a better definition. This paper proposes a new multi-axis modeling to better capture the temporal structure of events. In addition, we identify that event end-points are a major source of confusion in annotation, so we also propose to annotate TempRels based on start-points only. A pilot expert annotation using the proposed scheme shows significant improvement in IAA from the conventional 60's to 80's (Cohen's Kappa). This better-defined annotation scheme further enables the use of crowdsourcing to alleviate the labor intensity for each annotator. We hope that this work can foster more interesting studies towards event understanding.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Temporal Relation Annotation on English Temporal Annotation Resources](<https://www.sota2.com/research/sota/temporal-relation-annotation-on-english-temporal-annotation-resources.md>): Temporal Relation Annotation; English Temporal Annotation Resources; Event Count: 6099.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1804.07828>)

[PDF](<https://arxiv.org/pdf/1804.07828.pdf>)
