RI: EAGER: Collaborative Research: Adaptive Heads-up Displays for Simultaneous Interpretation

Project funded by the National Science Foundation: IIS-1748663 (UMD), IIS-1748642 (CMU)
PI: Graham Neubig, Carnegie Mellon University
PI: Hal Daumé III, University of Maryland
co-PI: Jordan Boyd-Graber, University of Maryland
co-PI: Leah Findlater, University of Washington

Overview

Interpretation, the task of translating speech from one language to another, is an important tool in facilitating communication in multi-lingual settings such as international meetings, travel, or diplomacy. However, simultaneous interpretation, during which the results must be produced as the speaker is speaking, is an extremely difficult task requiring a high level of experience and training. In particular, simultaneous interpreters often find certain content such as technical terms, names of people and organizations, and numbers particularly hard to translate correctly. This Early Grant for Exploratory Research project aims to create automatic interpretation assistants that will help interpreters with this difficult-to-translate content by recognizing this content in the original language, and displaying translations on a heads-up display (similar to teleprompter) for interpreters to use if they wish. This will make simultaneous interpretation more effective and accessible, making conversations across languages and cultures more natural, more common, and more effective and joining communities and cultures across the world in trade, cooperation, and friendship.

The major goal of the project is to examine methods for creating heads-up displays for simultaneous interpreters, providing real-time assistance with difficult-to-translate content. There are a number of goals for the project, including design, method development, and prototyping. These can be broken down into the following:

1. Create offline translation assistants: Create static aids that convey useful information to interpreters, automating the process of creating “cheat sheets”; given a short description of the material to be interpreted automatically build a lexicon specific to that domain. This includes discovering salient terms and finding translations for these terms.
2. Create machine-in-the-loop translation assistants: Create a display that will listen to to the speaker (in the source language), helping to create fluent translations, or possibly additionally the interpreter as well. The important thing in these interfaces is that they must not overwhelm the interpreter with irrelevant material.
3. Create methods for robust prediction: Noise manifests itself in the form of MT errors when using models on bilingual text, or ASR errors when using models on speech. In addition, there is incomplete input resulting from the inherently sequential process of interpretation, and models must be able to handle this.
4. Learning from explicit and implicit feedback: In order to create models that learn when and how to give suggestions to interpreters, we need a training signal about which suggestions are appropriate given a particular interpretation context. In order to do so, we can ask interpreters using the system to explicitly give feedback in real-time, or examine if forms of implicit feedback, which can be gleaned from observing user behavior in a deployed system.
5. Create initial design and elicit interpreter feedback: Perform participatory design sessions will consist of three components: (1) semi-structured interview questions on support needs during interpreting, (2) critique of mock-ups that explore a range of possible design elements (e.g., display type, size, and placement, type and amount of information displayed), and (3) an opportunity for participants to sketch or describe their own design enhancements.
6. Evaluations of the proposed interpretation interface: Deploy the system in a real interpretation setting and collect preliminary assessments with respect to objective measures of translation quality, the users’ subjective experience in using the system, and to measure cognitive load.

<< back to top

	Jordan Boyd-Graber Assistant Professor, Computer Science (Maryland)
	Hal Daumé III Professor, Computer Science (Maryland)
	Leah Findlater Associate Professor, Human Centered Design and Engineering (UW)
	Alvin Grissom II Assistant Professor, Computer Science (Ursinus)
	Graham Neubig Assistant Professor, Computer Science (CMU)
	Wenyan Li MS student, Electrical Engineering (Maryland)
	Denis Peskov Ph.D. student, Computer Science (Maryland)
	Jo Shoemaker Ph.D. student, Computer Science (Maryland)
	Craig Stewart MS student, Computer Science (CMU)
	Nikolai Vogler MS student, Computer Science (CMU)
	Chen Zhao Ph.D. student, Computer Science (Maryland)

<< back to top

• Craig Stewart, Nikolai Vogler, Junjie Hu, Jordan Boyd-Graber, and Graham Neubig. Automatic Estimation of Simultaneous Interpreter Performance. Association for Computational Linguistics, 2018. [Bibtex]
  
  

  @inproceedings{Stewart:Vogler:Hu:Boyd-Graber:Neubig-2018,
  	Title = {Automatic Estimation of Simultaneous Interpreter Performance},
  	Author = {Craig Stewart and Nikolai Vogler and Junjie Hu and Jordan Boyd-Graber and Graham Neubig},
  	Booktitle = {Association for Computational Linguistics},
  	Year = {2018},
  	Location = {Melbourne, Australia},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2018_acl_interpeval.pdf},
  }
  

• Nikolai Vogler, Craig Stewart, Graham Neubig. Lost in Interpretation: Predicting Untranslated Terminology in Simultaneous Interpretation Meeting of the North American Chapter of the Association for Computational Linguistics, 2019.
•  Vaibhav, Sumeet Singh, Craig Stewart, Graham Neubig. Improving Robustness of Machine Translation with Synthetic Noise. Meeting of the North American Chapter of the Association for Computational Linguistics, 2019.
• Denis Peskov, Joe Barrow, Pedro Rodriguez, Graham Neubig, and Jordan Boyd-Graber. Mitigating Noisy Inputs for Question Answering. Conference of the International Speech Communication Association, 2019. [Bibtex]
  
  

  @inproceedings{Peskov:Barrow:Rodriguez:Neubig:Boyd-Graber-2019,
  	Title = {Mitigating Noisy Inputs for Question Answering},
  	Author = {Denis Peskov and Joe Barrow and Pedro Rodriguez and Graham Neubig and Jordan Boyd-Graber},
  	Booktitle = {Conference of the International Speech Communication Association},
  	Year = {2019},
  	Location = {Graz, Austria},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2019_interspeech_asr},
  }
  

• Quality Estimation for Simultaneous Interpretation
• Automatic Prediction of Untranslated Terms

• Untranslated Term Annotations

• Alvin Grissom II and Jordan Boyd-Graber. Simultaneous Interpretation with Alvin Grissom II. ACL Simultaneous Machine Interpretation Workshop, 2020. [Bibtex]
  
  

  @online{Grissom-II:Boyd-Graber-2020,
  	Author = {Alvin {Grissom II} and Jordan Boyd-Graber},
  	Journal = {ACL Simultaneous Machine Interpretation Workshop},
  	Year = {2020},
  	Title = {Simultaneous Interpretation with Alvin Grissom II},
  	Url = {https://www.youtube.com/watch?v=ilaZNDHUhAM&t=2s},
  }
  

  
  
• Gino Dino. Terminology Assistance Coming to a Simultaneous Interpreter Near You. Slator, 2019. [Bibtex]
  
  

  @online{Dino-2019,
  	Author = {Gino Dino},
  	Journal = {Slator},
  	Year = {2019},
  	Title = {Terminology Assistance Coming to a Simultaneous Interpreter Near You},
  	Url = {https://slator.com/academia/terminology-assistance-coming-to-a-simultaneous-interpreter-near-you/},
  }
  

• Graham Neubig. Does Not Compute?. The Linguist, 2019. [Bibtex]
  
  

  @online{Neubig-2019,
  	Author = {Graham Neubig},
  	Journal = {The Linguist},
  	Year = {2019},
  	Title = {Does Not Compute?},
  	Url = {https://www.ciol.org.uk/the-linguist#ufh-i-497425041-the-linguist-58-1-feb-mar2019},
  }
  

• Esther Bond. Simultaneous Interpreters May Soon Get Real-Time Help Just When They Need It. Slator, 2018. [Bibtex]
  
  

  @online{Bond-2018,
  	Author = {Esther Bond},
  	Journal = {Slator},
  	Year = {2018},
  	Title = {Simultaneous Interpreters May Soon Get Real-Time Help Just When They Need It},
  	Url = {https://slator.com/academia/simultaneous-interpreters-may-soon-get-real-time-help-just-when-they-need-it/},
  }
  

Acknowledgments

This work is supported by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the researchers and do not necessarily reflect the views of the National Science Foundation.