Inter-annotator Agreement for Dependency Annotation of Learner Language

Marwa Ragheb and Markus Dickinson

Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications.

This paper reports on a study of inter-annotator agreement (IAA) for a dependency annotation scheme designed for learner English. Reliably-annotated learner corpora are a necessary step for the development of POS tagging and parsing of learner language. In our study, three annotators marked several layers of annotation over different levels of learner texts, and they were able to obtain generally high agreement, especially after discussing the disagreements among themselves, without researcher intervention, illustrating the feasibility of the scheme. We pinpoint some of the problems in obtaining full agreement, including annotation scheme vagueness for certain learner innovations, interface design issues, and difficult syntactic constructions. In the process, we also develop ways to calculate agreements for sets of dependencies.


Electronically available file formats:


Bibtex entry:

@InProceedings{ragheb:dickinson:13,
  author    = {Ragheb, Marwa  and  Dickinson, Markus},
  title     = {Inter-annotator Agreement for Dependency Annotation of 
               Learner Language},
  booktitle = {Proceedings of the Eighth Workshop on Innovative Use of 
               NLP for Building Educational Applications},
  year      = {2013},
  address   = {Atlanta, GA USA},
  pages     = {},
  url       = {http://cl.indiana.edu/~md7/papers/ragheb-dickinson13.html}
}