CAREER: Human-Computer Cooperation for Word-by-Word Question Answering

Project funded by the National Science Foundation (Originally IIS-1652666 at U Colorado, now 1822494 at UMD)
PI: Jordan Boyd-Graber, University of Maryland

This CAREER project investigates how humans and computers can work together to answer questions. Humans and computers possess complementary skills: humans have extensive commonsense understanding of the world and greater facility with unconventional language, while computers can effortlessly memorize countless facts and retrieve them in an instant. This proposal helps machines understand who people, places, and characters are; how to communicate this information to humans; and how to allow humans and computers to collaborate in question answering using limited information. A key component of this proposal is answering questions word-by-word: this forces both humans and computers to answer questions using information as efficiently as possible. In addition to embedding these skills in question answering tasks, this proposal has an extensive outreach program to exhibit this technology in interactive question answering competitions for high school and college students.

This research is possible by a new representations of entities in a medium-dimensional embedding that encodes relationships between entities (e.g., the representation of "Goodluck Jonathan" and "Nigeria" encodes that the former is the leader of the latter) to enable the system to answer questions about Nigeria. We validate the effectiveness of these representations both through traditional question answering evaluations and through interactive experiments with human collaboration to ensure that we can visualize these representations effectively. In addition to helping train computers to answer questions, we use opponent modeling and reinforcement learning to help train humans to better answer questions.

<< back to top

	Jordan Boyd-Graber Assistant Professor, Computer Science
	Ahmed Elgohary Ph.D. student, Computer Science
	Shi Feng Ph.D. student, Computer Science
	Pedro Rodriguez Ph.D. student, Computer Science
	Eric Wallace Undergrad, Computer Science
	Chen Zhao Ph.D. student, Computer Science

<< back to top

• Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh. Universal Adversarial Triggers for Attacking and Analyzing NLP, Emperical Methods in Natural Language Processing, 2019.
• Sahil Singla, Eric Wallace, Shi Feng, Soheil Feizi. Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation. International Conference on Machine Learning, 2019.
• Pedro Rodriguez, Shi Feng, Mohit Iyyer, He He, and Jordan Boyd-Graber. Quizbowl: The Case for Incremental Question Answering. ArXiv, Preprint. [Webpage] [Bibtex]
  
  

  @article{Rodriguez:Feng:Iyyer:He:Boyd-Graber-Preprint,
  	Title = {Quizbowl: The Case for Incremental Question Answering},
  	Author = {Pedro Rodriguez and Shi Feng and Mohit Iyyer and He He and Jordan Boyd-Graber},
  	Journal = {ArXiv},
  	Year = {Preprint},
  	Url = {https://arxiv.org/abs/1904.04792},
  }
  

• Jordan Boyd-Graber and Benjamin Börschinger. What Question Answering can Learn from Trivia Nerds. Association for Computational Linguistics, 2020. [Preprint] [Video] [Bibtex]
  
  

  @inproceedings{Boyd-Graber:B\"orschinger-2020,
  	Title = {What Question Answering can Learn from Trivia Nerds},
  	Author = {Jordan Boyd-Graber and Benjamin B\"orschinger},
  	Year = {2020},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2020_acl_trivia.pdf},
  	Location = {The Cyberverse Simulacrum of Seattle},
  	Booktitle = {Association for Computational Linguistics},
  }
  

  
  
• Tianze Shi, Chen Zhao, Jordan Boyd-Graber, Hal Daumé III, and Lillian Lee. On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries. Findings of EMNLP, 2020. [Data] [Preprint] [Bibtex]
  
  

  @inproceedings{Shi:Zhao:Boyd-Graber:Daume-III:Lee-2020,
  	Title = {On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries},
  	Author = {Tianze Shi and Chen Zhao and Jordan Boyd-Graber and Hal {Daum\'{e} III} and Lillian Lee},
  	Booktitle = {Findings of EMNLP},
  	Year = {2020},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2020_findings_qalign.pdf},
  }
  

• Chen Zhao, Chenyan Xiong, Xin Qian, and Jordan Boyd-Graber. Complex Factoid Question Answering with a Free-Text Knowledge Graph. The Web Conference, 2020. [Video] [Bibtex]
  
  

  @inproceedings{Zhao:Xiong:Qian:Boyd-Graber-2020,
  	Title = {Complex Factoid Question Answering with a Free-Text Knowledge Graph},
  	Author = {Chen Zhao and Chenyan Xiong and Xin Qian and Jordan Boyd-Graber},
  	Booktitle = {The Web Conference},
  	Year = {2020},
  	Location = {Taipei, Taiwan},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2020_www_delft.pdf},
  }
  

  
  
• Eric Wallace, Shi Feng, and Jordan Boyd-Graber. Misleading Failures of Partial-input Baselines. Association for Computational Linguistics, 2019. [Bibtex]
  
  

  @inproceedings{Wallace:Feng:Boyd-Graber-2019,
  	Title = {Misleading Failures of Partial-input Baselines},
  	Author = {Eric Wallace and Shi Feng and Jordan Boyd-Graber},
  	Booktitle = {Association for Computational Linguistics},
  	Year = {2019},
  	Location = {Florence, Italy},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2019_acl_flipside.pdf},
  }
  

• Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. Can You Unpack That? Learning to Rewrite Questions-in-Context. Empirical Methods in Natural Language Processing, 2019. [Data] [Bibtex]
  
  

  @inproceedings{Elgohary:Peskov:Boyd-Graber-2019,
  	Title = {Can You Unpack That? Learning to Rewrite Questions-in-Context},
  	Author = {Ahmed Elgohary and Denis Peskov and Jordan Boyd-Graber},
  	Booktitle = {Empirical Methods in Natural Language Processing},
  	Year = {2019},
  	Location = {Hong Kong, China},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2019_emnlp_sequentialqa.pdf},
  }
  

• Shi Feng and Jordan Boyd-Graber. What AI can do for me: Evaluating Machine Learning Interpretations in Cooperative Play. Intelligent User Interfaces, 2019. [Bibtex]
  
  

  @inproceedings{Feng:Boyd-Graber-2019,
  	Title = {What AI can do for me: Evaluating Machine Learning Interpretations in Cooperative Play},
  	Author = {Shi Feng and Jordan Boyd-Graber},
  	Booktitle = {Intelligent User Interfaces},
  	Year = {2019},
  	Location = {Los Angeles, CA},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2019_iui_augment.pdf},
  }
  

  
  
• Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples. Transactions of the Association of Computational Linguistics, 2019. [Code] [Videos] [Data] [Bibtex]
  
  

  @article{Wallace:Rodriguez:Feng:Yamada:Boyd-Graber-2019,
  	Title = {Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples},
  	Author = {Eric Wallace and Pedro Rodriguez and Shi Feng and Ikuya Yamada and Jordan Boyd-Graber},
  	Booktitle = {Transactions of the Association of Computational Linguistics},
  	Year = {2019},
  	Volume = {10},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2019_tacl_trick.pdf},
  }
  

  
  
• Eric Wallace and Jordan Boyd-Graber. Trick Me If You Can: Adversarial Writing of Trivia Challenge Questions. ACL Student Research Workshop, 2018. [Bibtex]
  
  

  @inproceedings{Wallace:Boyd-Graber-2018,
  	Title = {Trick Me If You Can: Adversarial Writing of Trivia Challenge Questions},
  	Author = {Eric Wallace and Jordan Boyd-Graber},
  	Booktitle = {ACL Student Research Workshop},
  	Year = {2018},
  	Location = {Melbourne, Australia},
  	Url = {http://aclweb.org/anthology/P18-3018},
  }
  

• Shi Feng, Eric Wallace, and Jordan Boyd-Graber. Interpreting Neural Networks with Nearest Neighbors. EMNLP Workshop on BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018. [Bibtex]
  
  

  @inproceedings{Feng:Wallace:Boyd-Graber-2018,
  	Title = {Interpreting Neural Networks with Nearest Neighbors},
  	Author = {Shi Feng and Eric Wallace and Jordan Boyd-Graber},
  	Booktitle = {EMNLP Workshop on BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP},
  	Year = {2018},
  	Location = {Brussels, Belgium},
  	Url = {http://aclweb.org/anthology/W18-5416},
  }
  

• Ahmed Elgohary, Chen Zhao, and Jordan Boyd-Graber. Dataset and Baselines for Sequential Open-Domain Question Answering. Empirical Methods in Natural Language Processing, 2018. [Bibtex]
  
  

  @inproceedings{Elgohary:Zhao:Boyd-Graber-2018,
  	Title = {Dataset and Baselines for Sequential Open-Domain Question Answering},
  	Author = {Ahmed Elgohary and Chen Zhao and Jordan Boyd-Graber},
  	Booktitle = {Empirical Methods in Natural Language Processing},
  	Year = {2018},
  	Location = {Brussels, Belgium},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2018_emnlp_linked.pdf},
  }
  

• Shi Feng, Eric Wallace, Alvin Grissom II, Pedro Rodriguez, Mohit Iyyer, and Jordan Boyd-Graber. Pathologies of Neural Models Make Interpretation Difficult. Empirical Methods in Natural Language Processing, 2018. [Blog Post] [Bibtex]
  
  

  @inproceedings{Feng:Wallace:II:Rodriguez:Iyyer:Boyd-Graber-2018,
  	Title = {Pathologies of Neural Models Make Interpretation Difficult},
  	Author = {Shi Feng and Eric Wallace and Alvin Grissom II and Pedro Rodriguez and Mohit Iyyer and Jordan Boyd-Graber},
  	Booktitle = {Empirical Methods in Natural Language Processing},
  	Year = {2018},
  	Location = {Brussels, Belgium},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2018_emnlp_rs.pdf},
  }
  

  
  
• Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Larry Davis. Learning to Color from Language. North American Association of Computational Linguistics, 2018. [Bibtex]
  
  

  @inproceedings{Iyyer:Manjunatha:Boyd-Graber:Davis-2018,
  	Title = {Learning to Color from Language},
  	Author = {Mohit Iyyer and Varun Manjunatha and Jordan Boyd-Graber and Larry Davis},
  	Booktitle = {North American Association of Computational Linguistics},
  	Year = {2018},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2018_naacl_colorization.pdf},
  }
  

• Jordan Boyd-Graber, Shi Feng, and Pedro Rodriguez. Human-Computer Question Answering: The Case for Quizbowl. The NIPS '17 Competition: Building Intelligent Systems, 2018. [Bibtex]
  
  

  @inbook{Boyd-Graber:Feng:Rodriguez-2018,
  	Editor = {Sergio Escalera and Markus Weimer},
  	Title = {Human-Computer Question Answering: The Case for Quizbowl},
  	Author = {Jordan Boyd-Graber and Shi Feng and Pedro Rodriguez},
  	Booktitle = {The NIPS '17 Competition: Building Intelligent Systems},
  	Publisher = {Springer Verlag},
  	Year = {2018},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2018_nips_qbcomp.pdf},
  }
  

• Leaderboard
• Question Answering System

• Data, competitions, source code

• Jordan Boyd-Graber, Pedro Rodriguez, and Pranav Goel. Four Jeopardy! champions competed against CMSC 470 Natural Language Processing Students' Question Answering Systems. 2019. [Bibtex]
  
  

  @online{Boyd-Graber:Rodriguez:Goel-2019,
  	Author = {Jordan Boyd-Graber and Pedro Rodriguez and Pranav Goel},
  	Year = {2019},
  	Title = {Four Jeopardy! champions competed against CMSC 470 Natural Language Processing Students' Question Answering Systems},
  	Location = {College Park, MD},
  	Url = {https://www.cs.umd.edu/article/2019/05/four-jeopardy-champions-competed-against-cmsc-470-natural-language-processing},
  }
  

  
  
• Matt Gardner and Waleed Ammar. Pathologies of Neural Models Make Interpretation Difficult. AI2 NLP Highlights, 2019. [Bibtex]
  
  

  @article{Gardner:Ammar-2019,
  	Author = {Matt Gardner and Waleed Ammar},
  	Year = {2019},
  	Title = {Pathologies of Neural Models Make Interpretation Difficult},
  	Journal = {AI2 NLP Highlights},
  	Url = {https://soundcloud.com/nlp-highlights/87-pathologies-of-neural-models-make-interpretation-difficult-with-shi-feng},
  }
  

• Kimbra Cutlip. Seeing How Computers "Think" Helps Humans Stump Machines and Reveals Artificial Intelligence Weaknesses. CMNS, 2019. [Bibtex]
  
  

  @online{Cutlip-2019,
  	Author = {Kimbra Cutlip},
  	Journal = {CMNS},
  	Year = {2019},
  	Title = {Seeing How Computers "Think" Helps Humans Stump Machines and Reveals Artificial Intelligence Weaknesses},
  	Url = {https://cmns.umd.edu/news-events/features/4470},
  }
  

• Sam Charrington. Pathologies of Neural Models and Interpretability with Alvin Grissom II. TWiML Talk, 2019. [Bibtex]
  
  

  @article{Charrington-2019,
  	Author = {Sam Charrington},
  	Year = {2019},
  	Title = {Pathologies of Neural Models and Interpretability with Alvin Grissom II},
  	Journal = {TWiML Talk},
  	Url = {https://twimlai.com/twiml-talk-229-pathologies-of-neural-models-and-interpretability-with-alvin-grissom-ii/},
  }
  

• Melissa Brachfeld. Boyd-Graber, Feng Present Paper on Machine Learning Interpretability at IUI 2019. UMIACS, 2019. [Bibtex]
  
  

  @online{Brachfeld-2019,
  	Author = {Melissa Brachfeld},
  	Journal = {UMIACS},
  	Year = {2019},
  	Title = {Boyd-Graber, Feng Present Paper on Machine Learning Interpretability at IUI 2019},
  	Url = {http://www.umiacs.umd.edu/about-us/news/boyd-graber-feng-present-paper-machine-learning-interpretability-iui-2019},
  }
  

• Matt Gardner and Waleed Ammar. A Discussion of Question Answering. AI2 NLP Highlights, 2018. [Bibtex]
  
  

  @article{Gardner:Ammar-2018,
  	Author = {Matt Gardner and Waleed Ammar},
  	Year = {2018},
  	Title = {A Discussion of Question Answering},
  	Journal = {AI2 NLP Highlights},
  	Url = {https://soundcloud.com/nlp-highlights/72-the-anatomy-question-answering-task-with-jordan-boyd-graber},
  }
  

• Matt Early Wright. Inside AI's "Black Box". Maryland Today, 2018. [Bibtex]
  
  

  @article{Wright-2018,
  	Author = {Matt Early Wright},
  	Year = {2018},
  	Title = {Inside AI's "Black Box"},
  	Journal = {Maryland Today},
  	Url = {https://today.umd.edu/articles/inside-ais-black-box-eda1240f-b360-45ac-9ee3-5d266b739f76},
  }
  

• Sala Levin. What Is ... a Research-Inspired Route to "Jeopardy!"?. Maryland Today, 2018. [Bibtex]
  
  

  @article{Levin-2018,
  	Author = {Sala Levin},
  	Year = {2018},
  	Title = {What Is ... a Research-Inspired Route to "Jeopardy!"?},
  	Journal = {Maryland Today},
  	Url = {https://today.umd.edu/articles/what-research-inspired-route-jeopardy-b2cfa4ae-4958-4b02-a574-e4b0517a464c},
  }
  

• Samir Singh. Summary of Pathologies of Neural Models Make Interpretations Difficult. UCI NLP, 2018. [Bibtex]
  
  

  @article{Singh-2018,
  	Author = {Samir Singh},
  	Year = {2018},
  	Title = {Summary of Pathologies of Neural Models Make Interpretations Difficult},
  	Journal = {UCI NLP},
  	Url = {https://medium.com/uci-nlp/summary-pathologies-of-neural-models-make-interpretations-difficult-emnlp-2018-62280abe9df8},
  }
  

• Brandi Adams. Associate Professor Jordan Boyd-Graber to appear on Jeopardy on September 26th 2018. UMD Computer Science, 2018. [Bibtex]
  
  

  @article{Adams-2018,
  	Author = {Brandi Adams},
  	Year = {2018},
  	Title = {Associate Professor Jordan Boyd-Graber to appear on Jeopardy on September 26th 2018},
  	Journal = {UMD Computer Science},
  	Url = {http://www.cs.umd.edu/article/2018/09/associate-professor-jordan-boyd-graber-appear-jeopardy-september-26th-2018},
  }
  

• Melissa Brachfeld. Boyd-Graber Publishes Paper in PNAS that Assesses Scholarly Influence. UMIACS, 2018. [Bibtex]
  
  

  @online{Brachfeld-2018,
  	Author = {Melissa Brachfeld},
  	Journal = {UMIACS},
  	Year = {2018},
  	Title = {Boyd-Graber Publishes Paper in PNAS that Assesses Scholarly Influence},
  	Url = {http://www.umiacs.umd.edu/about-us/news/boyd-graber-publishes-paper-pnas-assesses-scholarly-influence},
  }
  

Workshops Organized by Project Members

• Human-Computer Question Answering
• 2018 QANTA

Acknowledgments

This work is supported by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the researchers and do not necessarily reflect the views of the National Science Foundation.