Using semi-experts to derive judgments on word sense alignment: a pilot study

Soojeong Eom, Markus Dickinson, and Graham Katz

Proceedings of the 8th Language Resources and Evaluation Conference (LREC 2012). Istanbul, Turkey.

The overall goal of this project is to evaluate the performance of word sense alignment (WSA) systems, focusing on obtaining examples appropriate to language learners. Building a gold standard dataset based on human expert judgments is costly in time and labor, and thus we gauge the utility of using semi-experts in performing the annotation. In an online survey, we present a sense of a target word from one dictionary with senses from the other dictionary, asking for judgments of relatedness. We note the difficulty of agreement, yet the utility in using such results to evaluate WSA work. We find that one's treatment of related senses heavily impacts the results for WSA.


Electronically available file formats:


Bibtex entry:

@InProceedings{eom-et-al:12,
  author =       {Soojeong Eom, Markus Dickinson, and Graham Katz},
  title =        {Using semi-experts to derive judgments on word sense 
                  alignment: a pilot study},
  booktitle =    {Proceedings of the 8th Language Resources and 
                  Evaluation Conference (LREC 2012)},
  address =      {Istanbul, Turkey},
  pages =        {},
  url =          {\url{http://cl.indiana.edu/~md7/papers/eom-et-al12.html}},
  year =         {2012}
}