Schedule of Topics

This is the schedule for Seminar in Computational Linguistics: Topics in Computational Linguistics and the Cognitive Neuroscience of Language, Fall 2019

THIS SCHEDULE IS A WORK IN PROGRESS!
In addition, some topic areas may take longer than expected, so keep an eye on the class mailing list or e-mail me for "official" dates.

Readings will either be linked below, or will be available in the Linguistics Locker.

Overview, goals, administrivia; computation, cognition, and the brain

Background readings (optional)

• Marr, D. (1982), Vision: A Computational Approach, San Francisco, Freeman & Co., Introduction and Chapter 1 (ask me if you need the username and password to access this file)

Connecting levels of explanation

Debate involving levels of explanation and the relationships among them.

Readings

• Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A., & Poeppel, D. (2017). Neuroscience Needs Behavior: Correcting a Reductionist Bias. Neuron, 93(3), 480–490. https://doi.org/10.1016/j.neuron.2016.12.041

• Pater, J. (2019). Generative linguistics and neural networks at 60: Foundation, friction, and fusion. Language. https://doi.org/10.1353/lan.2019.0009

• Berent, I., & Marcus, G. (2019). No integration without structured representations: Response to Pater. Language. https://doi.org/10.1353/lan.2019.0011

• Dunbar, E. (2019). Generative grammar, neural networks, and the implementational mapping problem: Response to Pater. Language. https://doi.org/10.1353/lan.2019.0013

Also of interest

• Brain Inspired podcast. BI 025 John Krakauer: Understanding Cognition, Jan 25, 2019.

• Other responses to Pater in that issue of Language by Pearl, Linzen, and others.

• Pereira, F. (2000). Formal grammar and information theory: Together again?, Philosophical Transactions of the Royal Society of London, Series A, Mathematical, Physical and Engineering Sciences, 358(1769), 1239–1253. doi:10.1098/ rsta.2000.0583

Neuroanatomy and neuroimaging

Leader: Alexander

Shohini will lead in a lecture mode for a significant part of this session, making sure we are all up to speed on the fundamentals of language-relevant neuroanatomy and theories on language function and its localization. The Poldrack paper makes an essential methodological point about inferences connecting cognitive theories and brain activity.

Readings

• Ch. 1: The Human Brain. Kemmerer, D. (2015). Cognitive Neuroscience of Language. Psychology Press.

• Sections I to IV of Friederici, A. D. (2011). The brain basis of language processing: from structure to function. Physiol Rev, 91(4), 1357-1392. doi: 10.1152/physrev.00006.2011 -

• Russell A Poldrack, Can Cognitive Processes be Inferred from Neuroimaging Data?, Trends in Cognitive Sciences 10. no.2 (2006) 69-63, doi:10.1016/j.tics.2005.12.004.

Also of interest

• M.E. Raichle, A brief history of human brain mapping, Trends in Neurosciences, 2009.

• Ch. 2: Brain Mapping Methods. Kemmerer, D. (2015). Cognitive Neuroscience of Language. Psychology Press.

• Brain Science podcast. BS 156 with Russ Poldrack, Can fMRI Read Your Mind?

• Ioannidis, J. P. A. (2005). "Why Most Published Research Findings Are False". PLoS Medicine 2 (8): e124.

• Suzanne Dikker et al., MEG and Language, preprint, 2019. Submitted to an issue of Neuroimaging Clinics on Magnetoencephalography, Roland Lee and Mingxiong Huang (eds).

• Ch. 25: Neural Basis of Speech Perception. Hickok, G. and Poeppel, D. (2015). Neurobiology of Language. Academic Press.

Machine learning in neuroscience

Readings

• Glaser, Joshua I., Ari S. Benjamin, Roozbeh Farhoodi, and Konrad P. Kording. The roles of supervised machine learning in systems neuroscience. Progress in Neurobiology (2019).

  • Optional, but recommended especially for computational folks who want to play with the software: Glaser, Joshua I., Raeed H. Chowdhury, Matthew G. Perich, Lee E. Miller, and Konrad P. Kording. Machine learning for neural decoding., arXiv preprint arXiv:1708.00909 (2017); see accompanying package for neural decoding with many different machine learning techniques, https://github.com/KordingLab/Neural_Decoding.

• Carlson, T., Goddard, E., Kaplan, D. M., Klein, C., & Ritchie, J. B. (2018). Ghosts in machine learning for cognitive neuroscience: Moving from data to theory. NeuroImage, 180, 88-100.

• Toneva, M., & Wehbe, L. (to appear). Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). In Advances in Neural Information Processing Systems.

  • Also potentially of interest, once it's available: D. Schwartz, M. Toneva , and L. Wehbe, Inducing brain-relevant bias in natural language processing models, NeurIPS (to appear) [see abstract here]

Decoding (and Encoding)

Readings

• Murphy, Brian, Leila Wehbe, and Alona Fyshe. "Decoding language from the brain." Language, cognition, and computational models (2018): 53-80. (in PDF locker)

• Shailee Jain, Alexander Huth, Incorporating Context into Language Encoding Models for fMRI (2018)

• Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J. Gershman, Nancy Kanwisher, Matthew Botvinick & Evelina Fedorenko, Toward a universal decoder of linguistic meaning from brain activation (2018)

Also of interest

• Wehbe, L., Murphy, B., Talukdar, P., Fyshe, A., Ramdas, A., & Mitchell, T. (2014). Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PloS one, 9(11), e112575.

• Wehbe, L., Nunez-Elizalde, A. O., Huth, A. G., Deniz, F., Bilenko, N. Y., & Gallant, J. L. Deep multi-view representation learning of brain responses to natural stimuli. [Find longer version?]

• Huth, A. G., Lee, T., Nishimoto, S., Bilenko, N. Y., Vu, A. T., & Gallant, J. L. (2016). Decoding the semantic content of natural movies from human brain activity. Frontiers in systems neuroscience, 10, 81.

• Francisco Pereira, Tom Mitchell, Analysis of Brain Imaging Data With Machine Learning Methods (slides)

Encoding/Decoding, continued

Further discussion on decoding (and encoding). We have gotten most of the way through discussing Jain and Huth; we'll finish that discussion and also talk about Pereira et al. In addition, the recent episodes of Brain Inspired with David Poeppel seem very relevant to a lot of what we've been discussing and are a lot of fun, so we'll include those also.

• (Finish discussion of) Shailee Jain, Alexander Huth, Incorporating Context into Language Encoding Models for fMRI (2018)

• Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J. Gershman, Nancy Kanwisher, Matthew Botvinick & Evelina Fedorenko, Toward a universal decoder of linguistic meaning from brain activation (2018)

• Brain Inspired Podcast 046 David Poeppel: From Sounds to Meanings

• Brain Inspired Podcast 047 David Poeppel: From Sounds to Meanings

No class: Yom Kippur

Instead of the usual class, the class session time will be available for discussion of project ideas. (Shohini will be there but Philip will not.)

Looking at the word level

Progressing from lower level (recognition) up to individual-word level up to composition at phrase level

Readings

• Linzen, T., Marantz, A., & Pylkkänen, L. (2013). Syntactic context effects in visual word recognition: An MEG study. The Mental Lexicon, 8(2), 117-139. [MEG]

• Haoyan Xu, Brian Murphy, Alona Fyshe. BrainBench: A Brain-Image Test Suite for Distributional Semantic Models. Empirical Methods for Natural Language Processing, Austin, TX. 2016

• Alona Fyshe, Gustavo Sudr,e Leila Wehbe, Nicole Rafidi, and Tom M. Mitchell (2019). The Lexical Semantics of Adjective-Noun Phrases in the Human Brain. [preprint]

Language modeling and the time course of sentence processing

Readings

• Carried over from last time: Xu et al. (2019)

• Yan, Shaorong, and Aaron Steven White. "A Framework for Decoding Event-Related Potentials from Text", NAACL HLT 2019 (2019): 86.

• Ettinger, A., Feldman, N., Resnik, P. and Phillips, C., 2016. Modeling N400 amplitude using vector space models of word representation. In CogSci.

Optional background

• Armeni, K., Willems, R. M., & Frank, S. L. (2017). Probabilistic language models in cognitive neuroscience: Promises and pitfalls. Neuroscience & Biobehavioral Reviews, 83, 579-588.

Hierarchy from a cognitive/neuro angle

Readings

• Ettinger et al. (2016) (from last time)

• Ding, Nai, Lucia Melloni, Xing Tian and David Poeppel. Rule-based and Word-level Statistics-based Processing of Language: Insights from Neuroscience. Language, cognition and neuroscience 32 5 (2017): 570-575 .

• Frank, S. L., & Christiansen, M. H. (2018). Hierarchical and sequential processing of language a response to: Ding, melloni, tian, and poeppel (2017). rule-based and word-level statistics-based processing of language: Insights from neuroscience. language, cognition and neuroscience. Language, Cognition and Neuroscience. https://doi.org/10.1080/23273798.2018.1424347

• Nelson, M. J., El Karoui, I., Giber, K., Yang, X., Cohen, L., Koopman, H., ... & Dehaene, S. (2017). Neurophysiological dynamics of phrase-structure building during sentence processing. Proceedings of the National Academy of Sciences, 114(18), E3669-E3678.

  • For reference, the early work: Hale, J. (2001). A probabilistic Earley parser as a psycholinguistic model. Paper presented at the Proceedings of the second meeting of the North American chapter of the Association for Computational Linguistics, Pittsburgh, PA. doi:10.3115/ 1073336.1073357

Also of interest as background:

• Ding, N., Melloni, L., Zhang, H., Tian, X., & Poeppel, D. (2016). Cortical tracking of hierarchical linguistic structures in connected speech. Nature Neuroscience. doi:10.1038/nn. 4186

Hierarchy in distributed representations

The topic of discussion here is the extent to which neural networks might capture syntactic/hiherarchical structure without explicit building-in of hierarchical mechanisms. There is a cottage industry of papers on this topic; see the Yoav Goldberg arXiv report for a very concrete comparison of a number of approaches.

Readings

• Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., & Baroni, M. (2018). Colorless Green Recurrent Networks Dream Hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (pp. 1195–1205). Stroudsburg, PA, USA: Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-1108

• John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan Brennan. 2018. Finding syntax in human encephalography with beam search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2727–2736. Association for Computational Linguistics.

• Richard Futrell, Ethan Wilcox, Takashi Morita, Miguel Ballesteros, Roger Levy. Neural Language Models as Psycholinguistic Subjects: Representation of Syntactic State. NAACL-HLT 2019

• Goldberg, Yoav. Assessing BERT's Syntactic Abilities. arXiv preprint arXiv:1901.05287 (2019).

• Wolf, Thomas. Some additional experiments extending the tech report “Assessing BERT’s Syntactic Abilities” by Yoav Goldberg. Technical report, 2019.

• Elman (1990)

• Linzen et al. (2016)

• Hewitt and Manning (2019)

Neuroimaging and natural stimuli: methodological discussion

Readings

• Willems, R. M. (2015). Introduction. In Cognitive Neuroscience of Natural Language Use. Cambridge, England: Cambridge University Press. Cognition and Neuroscience. https://doi.org/10.1080/23273798.2017.1323110 [In PDF locker]

• Alday, P. M. (2019). M/EEG analysis of naturalistic stories: a review from speech to language processing. Language, Cognition and Neuroscience, 34(4), 457-473. [in PDF locker]

• Brennan, J. (2016). Naturalistic Sentence Comprehension in the Brain. Language and Linguistics Compass. https://doi.org/10.1111/lnc3.12198

Also of interest:

• Hamilton, L. S., & Huth, A. G. (2018). The revolution will not be controlled: natural stimuli in speech neuroscience. Language, Cognition and Neuroscience. https://doi.org/10.1080/23273798.2018.1499946

• Rodd, J. M., & Davis, M. H. (2017). How to study spoken language understanding: a survey of neuroscientific methods. nbLanguage,

• Kandylaki, Katerina D., and Ina Bornkessel-Schlesewsky. "From story comprehension to the neurobiology of language." (2019): Journal Language, Cognition and Neuroscience, 34, 405-410.

• Include as optional background: something for Christian?

Studies using natural stimuli

Readings:

• Bhattasali, S., Fabre, M., Luh, W.-M., Al Saied, H., Constant, M., Pallier, C., … Hale, J. (2018). Localising memory retrieval and syntactic composition: an fMRI study of naturalistic language comprehension. Language, Cognition and Neuroscience, 1–20. doi: 10.1080/23273798.2018.1518533

• Brodbeck, C., Hong, L. E., & Simon, J. Z. (2018). Rapid Transformation from Auditory to Linguistic Representations of Continuous Speech. Current Biology, 28(24), 3976-3983.e5. https://doi.org/10.1016/J.CUB.2018.10.042

Carried over from last week, this paper will be treated as background, with relevant aspects of the paper incorporated into this week's discussion if/as appropriate:

• Brennan, J. (2016). Naturalistic Sentence Comprehension in the Brain. Language and Linguistics Compass. https://doi.org/10.1111/lnc3.12198

Also of interest:

• Broderick, Michael P., Andrew J. Anderson, Giovanni M. Di Liberto, Michael J. Crosse, and Edmund C. Lalor. "Electrophysiological correlates of semantic dissimilarity reflect the comprehension of natural, narrative speech". Current Biology 28, no. 5 (2018): 803-809.

• Abstract Linguistic Structure Correlates with Temporal Activity during Naturalistic Comprehension, Jonathan R. Brennan, Edward P. Stabler, Sarah E. Van Wagenen, Wen-Ming Luh, John Hale. Brain and Language, volume 157—158 June–July 2016 pages 81-94. http://dx.doi.org/10.1016/j.bandl.2016.04.008. Already peer-reviewed, closely related to Li & Hale chapter.

Happy Thanksgiving!

The predictive brain

• Clark, Andy. "Whatever next? Predictive brains, situated agents, and the future of cognitive science". Behavioral and brain sciences 36, no. 3 (2013): 181-204.

  • Also note: "A exceptionally large number of excellent commentary proposals inspired a special research topic for further discussion of this target article’s subject matter, edited by Axel Cleeremans and Shimon Edelman in Frontiers in Theoretical and Philosophical Psychology. This discussion has a preface by Cleeremans and Edelman and 25 commentaries and includes a separate rejoinder from Andy Clark. See: http://www.frontiersin.org/Theoretical_and_Philosophical_Psychology/researchtopics/Forethought_as_an_evolutionary/1031")

• Shain, C., Blank, I. A., van Schijndel, M., Schuler, W., & Fedorenko, E. (2019). fMRI reveals language-specific predictive coding during naturalistic sentence comprehension. bioRxiv, 717512.

• Heeger - Theory of Cortical Function. [Prediction is constantly taking place, but he models things as a dynamical system that is constantly optimizing a loss function that balances top down prediction, bottom-up evidence, and what he calls "prior drive". The loss function also includes parameters intended to capture the tradeoff between top-down and bottom-up so that, for example, external factors like attention can adjust for how you trade off between the two.]

Assortment of other related topics we didn't get to

• Friston - free energy principle, Bayesian brain hypothesis; Bayesian pragmatics applied to brain?

• Bornkessel-Schlesewsky, I., & Schlesewsky, M. (2013). Reconciling time, space and function: a new dorsal–ventral stream model of sentence comprehension. Brain and language, 125(1), 60-76.

• Friederici, A. D. (2012). The cortical language circuit: from auditory perception to sentence comprehension. Trends in cognitive sciences, 16(5), 262-268.

• Hagoort, P., & Indefrey, P. (2014). The neurobiology of language beyond single words. Annual review of neuroscience, 37, 347-362

• Complementary Learning Systems (Kumaran et al. 2016) -- relating non-statistical and statistical learning (hippocampus and cortex)

  • Mirković, J., Vinals, L. and Gaskell, M.G., 2019. The role of complementary learning systems in learning and consolidation in a quasi-regular domain. Cortex, 116, pp.228-249. (See also other articles on morphological processing in the brain at https://www.sciencedirect.com/journal/cortex/vol/116/suppl/C.)

• 2016. Temporal Lobes as Combinatory Engines for both Form and Meaning. Jixing Li, Jonathan Brennan, Adam Mahar, John Hale.. Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), Osaka, Japan. Conference-paper sized writeup of Jixing Li’s work aimed at computational linguists. Also look at more recent modeling - Chinese as well as English?

• Hickok, G. (2012). Computational neuroanatomy of speech production. Nature Reviews Neuroscience. https://doi.org/10.1038/nrn3158 - add a day looking at computational modeling for language production and comprehension, feedback?

• Gwilliams, L., Linzen, T., Poeppel, D., & Marantz, A. (2018). In spoken word recognition, the future predicts the past. The Journal of Neuroscience, 38(35), 7585–7599. DOI: https://doi.org/10.1101/150151

• 2015. Modeling fMRI time courses with linguistic structure at various grain sizes David E. Lutz, Wen-Ming Luh, Jonathan R. Brennan, John Hale. Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, North American Association for Computational Lingustics. Denver, CO. [This is a shorter writeup of the same project reported in the Brain and Language article.]

• Looking at the discourse level -- ?

Philip Resnik, Associate Professor
Department of Linguistics and Institute for Advanced Computer Studies


Department of Linguistics
1401 Marie Mount Hall            UMIACS phone: (301) 405-6760       
University of Maryland           Linguistics phone: (301) 405-8903
College Park, MD 20742 USA	   Fax: (301) 314-2644 / (301) 405-7104
http://umiacs.umd.edu/~resnik	   E-mail: resnik AT umd _DOT.GOES.HERE_ edu