Evaluating Parse Error Detection across Varied Conditions

Amber Smith and Markus Dickinson

Proceedings of the 13th International Workshop on Treebanks and Linguistic Theories (TLT13).

We investigate parse error detection methods under real-world conditions, outlining and testing different variables for evaluation and pointing to useful experimental practices. In particular, we focus on four different conversion methods, ten different training data sizes, two parsers, and three error detection methods. By comparing a set number of tokens across conditions, we measure error detection precision and revised labeled attachment scores to see the effect of each of the variables. We show the interactions between variables and the importance of accounting for parser choice and training data size (cf. initial parser quality). Importantly, we provide a framework for evaluating error detection and thus helping build large annotated corpora.


Electronically available file formats:


Bibtex entry:

@InProceedings{smith:dickinson:14,
  author    = {Smith, Amber and Dickinson, Markus},
  title     = {Evaluating Parse Error Detection across Varied Conditions},
  booktitle = {Proceedings of the 13th International Workshop on 
               Treebanks and Linguistic Theories (TLT13},
  year      = {2014},
  address   = {T\"ubingen, Germany},
  pages     = {230--241},
  url       = {http://cl.indiana.edu/~md7/papers/smith-dickinson14.html}
}