L645 / B659
Advanced Natural Language Processing (NLP)

Fall 2015

Course goals In recent years, statistical methods have become the standard in the field of Natural Language Processing (NLP). This course gives an introduction to statistical models and machine learning paradigms in NLP. Such methods are helpful for reaching wide coverage, reducing ambiguity, automatic learning, increasing robustness, etc.

In this course, we will cover basic notions in probability and information theory, focusing on the concepts needed for NLP. Then we will discuss (Hidden) Markov Models, exemplified by an approach to POS tagging. The following sessions will be dedicated to probabilistic approaches to parsing, focusing on probabilistic context-free grammars.

Additionally, we will cover topics such as word sense disambiguation, text categorization, and statistical alignment methods and their use in machine translation. What we cover in the last third of the class will depend in part upon student interest. We will be focusing on statistical methods in the context of particular tasks, but all of the methods we will use are applicable to a range of tasks in NLP. Thus, this course provides an essential platform for further work in NLP.

Meeting time: MW, 1:00–2:15pm

Classroom: Swain East (SE) 245

Course website: http://cl.indiana.edu/~md7/15/645/

Assignments, slides, etc. will be posted here. (I only use oncourse for emails & semi-restricted data.)

Credits: 3

Recommended course prerequisites: Some knowledge of either linguistics or computer science is extremely beneficial, with Computation and Linguistic Analysis (L545) being the strongest background. Some programming experience is expected.

Instructor: Markus Dickinson

Office: Memorial Hall (MM) 317

Phone: 856-2535

E-mail: md7@thegrandbudapestindiana.edu (remove the stuff worth removing)

Office hours:

R 11:00am–12:00pm
or by appointment


Course requirements:

Additional Opportunities I would encourage you to check out the weekly colloquium series on Computational Linguistics (CLingDing): see http://cl.indiana.edu/wiki/. Note, too, that there is an email listserv (COMPLING-L) which has announcements for talks, internships, etc.

Also, as it so happens, the Workshop on Methods will be bringing in David Blei (on October 16, 2–4pm), just as we will be about to start discussing topic modeling. I encourage you to attend. More information is at: http://ssrc.indiana.edu/seminars/wim.shtml

Academic Integrity (from the Dean for Academic Standards and Opportunities): As a student at IU, you are expected to adhere to the standards and policies detailed in the Code of Student Rights, Responsibilities, and Conduct (http://www.iu.edu/~code/). When you submit an assignment with your name on it, you are signifying that the work contained therein is all yours, unless otherwise cited or referenced. Any ideas or materials taken from another source for either written or oral use must be fully acknowledged. If you are unsure about the expectations for completing an assignment or taking a test or exam, be sure to seek clarification beforehand. All suspected violations of the Code will be handled according to University policies. Sanctions for academic misconduct may include a failing grade on the assignment, reduction in your final course grade, a failing grade in the course, among other possibilities, and must include a report to the Dean of Students who may impose additional disciplinary sanctions.

Students with Disabilities: Students who need an accommodation based on the impact of a disability should contact me to arrange an appointment as soon as possible to discuss the course format, to anticipate needs, and to explore potential accommodations.

I rely on Disability Services for Students for assistance in verifying the need for accommodations and developing accommodation strategies. Students who have not previously contacted Disability Services are encouraged to do so (812-855-7578; http://www.indiana.edu/~iubdss/).

(Tentative) Schedule:




Aug. 24Intro to class (.pdf, 2x3.pdf)

MS, ch. 1

26Probability Theory (.pdf, 2x3.pdf)

MS, 2.1

31Probability Theory

KS, 1.1–1.4

Sep. 2Information Theory (.pdf, 2x3.pdf)

MS, 2.2; KS, 2.2

HW1 due

7Labor Day, no classes

9Corpora and Linguistic Annotation (.pdf, 2x3.pdf)

MS, ch. 3, 4

14Markov Chains (.pdf, 2x3.pdf)

MS, 9.1

HW2 due
16Markov Models

KS, 2.1.1–2.1.3

21N-gram POS tagging (.pdf, 2x3.pdf)

23Smoothing (.pdf, 2x3.pdf)

JM, 4.5; MS, 6.2

HW3 due

28Hidden Markov Models (.pdf, 2x3.pdf)

MS, 9.2–9.3; KS, 2.1.4

30Calculating P(O) (.pdf, 2x3.pdf)

MS, 9.3.1; KS, 2.1.5

HW4 due

Oct. 5Finding the Optimal State Sequence

MS, 9.3.2; KS, 2.1.6

7Parameter Estimation (.pdf, 2x3.pdf)

MS, 9.3.3; KS, 2.1.7

12Topic Modeling (tutorial slides)

Blei (2012)

HW5 due
14Topic Modeling

Blei & Lafferty (2009)

16[David Blei talk, SSRC, 2–4pm]

19Sentiment Analysis (tutorial slides)

Liu (2010)

HW6 due
21Sentiment Analysis (handout)

26CYK parsing (.pdf, 2x3.pdf)

JM, 13–13.4.1

28Probabilistic Context-Free Grammars (.pdf, 2x3.pdf)

MS, 11.1–11.3.3

HW7 due

Nov. 1PCFGs (.pdf, 2x3.pdf)

JM, 14.2, 14.4–14.5

3Probabilistic Parsing

MS, ch. 12

9Estimating PCFGs (.pdf, 2x3.pdf)

MS, 11.3.4–11.4

HW8 due
11Estimating PCFGs

16Practical Parsing (.pdf, 2x3.pdf)

18Word Sense Disambiguation (WSD) (.pdf, 2x3.pdf)

MS, ch. 7

HW9 due

23Thanksgiving break, no classes

25Thanksgiving break, no classes

30Deep learning (tutorial slides)

Mikolov et al (2013)

Dec. 2Deep learning

Levy et al (2015)

7Semantic role labeling (SRL)


HW10 due


16Final HW/Project due

Final HW due

Disclaimer This syllabus is subject to change and likely will change. All important changes will be made in writing, with ample time for adjustment.