Schedule of Topics

This is the schedule of topics for Seminar on Computational Modeling of Language, Fall 2010.

THIS SCHEDULE IS A WORK IN PROGRESS!
In addition, some topic areas may take longer than expected, so keep an eye on the class mailing list or e-mail me for "official" dates.

• September 1. Administrivia, group collaboration and discussion tools, overview of course. What is a "computational model of language" from the perspective this class?
  • Recommended: Sagan (1979), Can we Know the Universe?
  • Recommended: Resnik (1992), Left-Corner Parsing and Psychological Plausibility, Proceedings of the Fourteenth International Conference on Computational Linguistics (COLING '92), Nantes, France, 1992.

• September 8. Information theoretic modeling: surprisal, search, and syntax
  • Recommended: Sections 1-2 of Hale, J. (to appear), What a rational parser would do. To appear in Cognitive Science.
  • Hale, J. A Probabilistic Earley Parser as a Psycholinguistic Model. In Proceedings of the Second Meeting of the North American Chapter of the Asssociation for Computational Linguistics.
  • Boston, M. F., Hale, J., Kliegl, R., Patil, U. & Vasishth, S. (2008). Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam Sentence Corpus. Journal of Eye Movement Research, 2(1):1, 1-12.

• September 15. Guest: Roger Levy. Information theoretic modeling.

  Note: Roger will also be giving a talk at 11am in AV Williams 2120 in the Computational Linguistics Colloquium Series, and he will be available for meetings on both Tuesday and Wednesday. Contact me if you'd like to meet with him. (The two discussions will be related, but each will be accessible without attending the other.)

  • Levy, R. (EMNLP 2008), A noisy-channel model of rational human sentence comprehension under uncertain input
  • Bicknell & Levy, (ACL 2010), A Rational Model of Eye Movement Control in Reading. A reinforcement-learning approach to modeling eye-movement control.
  • Optional: Levy, Bicknell, Slattery, & Rayner (PNAS 2009), Eye movement evidence that readers maintain and act on uncertainty about past linguistic input. This presents an eye-tracking experiment whose results validate predictions of the uncertain-input model in the EMNLP 2008 reading. Roger comments that reading this and the EMNLP paper together provides a good picture of his approach to presenting the same research program to both computational linguistics audiences and psycholinguists/cognitive scientists.

• September 22 Information theoretic modeling: uniform information density
  • Frank, A. and Jaeger, T.F. 2008. Speaking Rationally: Uniform Information Density as an Optimal Strategy for Language Production. The 30th Annual Meeting of the Cognitive Science Society (CogSci08), 933-938.
  • Jaeger, T.F. 2010. Redundancy and reduction: Speakers manage syntactic information density. Cognitive Psychology 61(1), 23-62.

• September 29 Non-parametric Bayesian models
  
  • Optional (review of parametric modeling concepts and inference via sampling): Philip Resnik and Eric Hardisty, Gibbs Sampling for the Uninitiated. Technical report, April 2010. University of Maryland techreport numbers CS-TR-4956, UMIACS-TR-2010-04, LAMP-153.
  • Sam Gershman and Dave Blei: Bayesian Nonparametric Models: A Tutorial. (This is the updated version, revising prior to publication in Journal of Mathematical Psychology. Comments for the authors are welcome.)
  • S. Williamson, C. Wang, K. Heller, and D. Blei. The IBP compound Dirichlet process and its application to focused topic modeling. International Conference on Machine Learning, 2010.

• October 6 More nonparametrics: Hierarchical Dirichlet Processes
  
  • S. Williamson, C. Wang, K. Heller, and D. Blei. The IBP compound Dirichlet process and its application to focused topic modeling. International Conference on Machine Learning, 2010.
  • Griffiths et al., Unifying Rational Models of Categorization via the Hierarchical Dirichlet Process
  • Optional (details on HDP, recommend at least looking this over): Teh, Jordan, Beal, and Blei, Hierarchical Dirichlet Processes.

• October 13 Guest: Jordan Boyd-Graber. Adaptor Grammars.
  
  • Mark Johnson and Thomas Griffiths, Adaptor Grammars: A Framework for Specifying Compositional Nonparametric Bayesian Models.
  • Mark Johnson, PCFGs, Topic Models, Adaptor Grammars and Learning Topical Collocations and the Structure of Proper Names
  • Optional: Eric Hardisty, Jordan Boyd-Graber, and Philip Resnik. Modeling Perspective using Adaptor Grammars. Empirical Methods in Natural Language Processing, 2010.

• October 13 Structured models
  • Kevin Knight, Bayesian Inference with Tears: a tutorial workbook for natural language researchers, September 2009.
  • Mark Johnson, Thomas L. Griffiths and Sharon Goldwater (2007), Bayesian Inference for PCFGs via Markov Chain Monte Carlo, Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 139-146.

• October 27 Guest: Hal Daumé
  • Hal Daumé III and Lyle Campbell, A Bayesian Model for Discovering Typological Implications, Conference of the Association for Computational Linguistics (ACL), 2007.
  • Adam R. Teichert and Hal Daumé III, Unsupervised Part of Speech Tagging Without a Lexicon

• November 3 LDA+
  • Limin Yao, David Mimno and Andrew McCallum. Efficient Methods for Topic Model Inference on Streaming Document Collections. Conference on Knowledge Discovery and Data Mining (KDD), 2009, Paris, France.
  • Michael Paul and Roxana Girju. A Two-Dimensional Topic-Aspect Model for Discovering Multi-Faceted Topics. In proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI-10), Atlanta, Georgia. July 2010.
  • Optional: Michael J. Paul, ChengXiang Zhai and Roxana Girju. Summarizing Contrastive Viewpoints In Opinionated Text. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP 2010), MIT, Cambridge, Massachusetts. October 2010.

• November 10 Guest: Bob Carpenter.
  • Bob Carpenter, A Multilevel Bayesian Model of Categorical Data Annotation, unpublished manuscript, 2009.

  • Additional readings worth looking at:
    • Widely cited early paper on this topic: A. P. Dawid, and A. M. Skene, Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm, Applied Statistics, Vol. 28, Nr. 1 (1979), p. 20--28.
    • Brief but relevant background discussion and pointers on Panos Ipeirotis's blog, focused in particular on these ideas as applied to quality control on Mechanical Turk.
    • More discussion and background at the end of Bob's LREC tutorial slides.
    • Bob's analyses of more data sets with a wider range of models and extensive references can be found in his tech report.

• November 17 Project discussion

  Each person or team should spend, say, 20 minutes or so describing and dicussing their project. Informally is fine -- no need for slides. (If someone wants to present slides, let me know in advance so I can make sure there's a projector.) Presentations should include the basic elements you gave me in the project proposal (basic goals/motivations, data you're using, evaluation plan, etc.) and a status report together with any obstacles, challenges, or worries. (I'm hoping if you're hitting any real obstacles, you'll be telling me sooner rather than later!) I'm hoping that this will lead to discussion that is useful to everyone, sharing of ideas, etc. Presenters can decide if they want to take interruptions, or clarification questions only for the first half followed by a question and discussion period. I have no objections if this leads to sharing of data and/or code, as long as everyone's contributions are clearly identified in project writeups.

  To turn in: A brief summary of everyone else's projects. For each one, include:

  • The people doing the project.
  • Your own brief summary of what the project is about.
  • What you think the likelihood of an interesting outcome is, and why.
  • What you think the most important challenges or difficulties are, and why.
  • Any other comments for me or the people doing the project.

• November 24: No class. Have a great Thanksgiving!

• December 1 Guest: Naomi Feldman
  • Shi, L., Griffiths, T. L., Feldman, N. H, & Sanborn, A. N., 2010, Exemplar models as a mechanism for performing Bayesian inference, Psychonomic Bulletin & Review, 17, 443-464. (That's a link to the long-form version of the paper, doublespaced and 58 pages long. The published version is available from the journal Web site, or ask me to e-mail it to you.)
  • Roger Levy, Florencia Reali, and Thomas Griffiths. 2009. Modeling the effects of memory on human online sentence processing with particle filters. Proceedings of the Twenty-second Annual Conference on Neural Information Processing Systems.

• December 8 Project presentations