Closing the User-Model Loop for Understanding Topics in Large Document Collections

Project funded by the National Science Foundation (IIS-1409287, UMD; IIS-1409739, BYU)
PI: Jordan Boyd-Graber (Maryland), co-PI: Leah Findlater (UW), PI: Kevin Seppi (BYU)

Individuals and organizations must cope with massive amounts of unstructured text information: individuals sifting through a lifetime of e-mail and documents, journalists understanding the activities of government organizations, companies reacting to what people say about them online, or scholars making sense of digitized documents from the ancient world. This project’s research goal is to bring together two previously disconnected components of how users understand this deluge of data: algorithms to sift through the data and interfaces to communicate the results of the algorithms. This project allows users to provide feedback to algorithms that were typically employed on a “take it or leave it” basis: if the algorithm makes a mistake or misunderstands the data, users can correct the problem using an intuitive user interface and improve the underlying analysis. This project jointly improves both the algorithms and the interfaces, leading to deeper understanding of, faster introduction to, and greater trust in the algorithms we rely on to understand massive textual datasets. Furthermore, source code and functional demos will be shared publicly, and tutorials will be shared online and in person in to aid the adoption of the methodologies.

This project enables computer algorithms and humans each to apply their strengths and collaborate in managing and making sense of large volumes of textual data. It “closes the loop” in novel ways to connect users with a class of big data analysis algorithms called topic models. This connection is made through interfaces that empower the user to change the underlying models by refining the number and granularity of topics, adding or removing words considered by the model, and adding constraints on what words appear together in topics. The underlying model also enables new visualizations in the form of a Metadata Map that uses active learning to focus users’ limited attention on the most important documents in a collection. Users annotate documents with useful meta-data and thereby further improve the quality of the discovered topics. The project includes evaluations of these methods through careful user studies and in-depth case studies to demonstrate that topics are more coherent, users can more quickly provide annotations, users trust the underlying algorithms more, and users can more effectively build an understanding of their textual data.

<< back to top

	Jordan Boyd-Graber Associate Professor, Computer Science/Language Science/iSchool/UMIACS (Maryland)
	Leah Findlater Assistant Professor, Human Centered Design and Engineering (UMD)
	Kevin Seppi Professor, Computer Science (BYU)
	Piper Armstrong MS Student, Computer Science (BYU)
	Stephen Cowley MS Student, Computer Science (BYU)
	Wilson Fearn MS Student, Computer Science (BYU)
	Courtni Byun MS Student, Computer Science (BYU)
	Fenfei Guo PhD Student, Computer Science (UMD)
	Varun Kumar Applied Scientist at Amazon
	Pedro Rodriguez Ph.D. student, Computer Science (Colorado)
	Alison Smith-Renner PhD Student, Computer Science (UMD)
	Thang Nguyen PhD Student, Computer Science (UMD)

<< back to top

	Eric Ringger Associate Professor, Computer Science (BYU) Now Senior Director of Maching Learning for Personalization at Zillow
	Paul Felt PhD Student, Computer Science (BYU) Now Software Engineer at IBM
	Ethan Garofolo MS Student, Computer Science (BYU)
	Jeff Lund PhD Student, Computer Science (BYU) Now at Google
	Connor Cook Undergrad, Computer Science (BYU) Now an MS Student at the University of Colorado Boulder
	Tak Yeon Lee PhD Student, Computer Science (UMD) Now Research Scientist at Adobe
	You Lu MS Student, Computer Science (Colorado) Now PhD Student at Virginia Tech
	Viet-An Nguyen PhD Student, Computer Science (UMD) Now Research Scientist at Facebook
	Nozomu Okuda MS Student, Computer Science (BYU)
	Forough Poursabzi PhD Student, Computer Science (Colorado) Now Postdoc at MSR NYC

<< back to top

• Alison Smith, Jordan Boyd-Graber, Ron Fan, Melissa Birchfield, Tongshuang Wu, Dan Weld, and Leah Findlater. No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML. Computer-Human Interaction, 2020. [Bibtex]
  
  

  @inproceedings{Smith:Boyd-Graber:Fan:Birchfield:Wu:Weld:Findlater-2020,
  	Author = {Alison Smith and Jordan Boyd-Graber and Ron Fan and Melissa Birchfield and Tongshuang Wu and Dan Weld and Leah Findlater},
  	Booktitle = {Computer-Human Interaction},
  	Year = {2020},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2020_chi_explanation.pdf},
  	Title = {No Explainability without Accountability: An Empirical Study of Explanations and Feedback in Interactive ML},
  }
  

• Alison Smith, Varun Kumar, Jordan Boyd-Graber, Kevin Seppi, and Leah Findlater. Digging into User Control: Perceptions of Adherence and Instability in Transparent Models. Intelligent User Interfaces, 2020. [Bibtex]
  
  

  @inproceedings{Smith:Kumar:Boyd-Graber:Seppi:Findlater-2020,
  	Author = {Alison Smith and Varun Kumar and Jordan Boyd-Graber and Kevin Seppi and Leah Findlater},
  	Booktitle = {Intelligent User Interfaces},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2020_iui_control.pdf},
  	Year = {2020},
  	Title = {Digging into User Control: Perceptions of Adherence and
  Instability in Transparent Models},
  }
  

• Fenfei Guo, Jordan Boyd-Graber, Mohit Iyyer, and Leah Findlater. Which Evaluations Uncover Sense Representations that Actually Make Sense?. Linguistic Resources and Evaluation Conference, 2020. [Bibtex]
  
  

  @inproceedings{Guo:Boyd-Graber:Iyyer:Findlater-2020,
  	Author = {Fenfei Guo and Jordan Boyd-Graber and Mohit Iyyer and Leah Findlater},
  	Location = {France (but only in dreams)},
  	Booktitle = {Linguistic Resources and Evaluation Conference},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2020_lrec_sense.pdf},
  	Year = {2020},
  	Title = {Which Evaluations Uncover Sense Representations that Actually Make Sense?},
  }
  

• Jeffrey Lund, Piper Armstrong, Wilson Fearn, Stephen Cowley, Emily Hales and Kevin Seppi. Cross-referencing Using Fine-grained Topic Modeling. North American Association for Computational Linguistics. Minneapolis, MN. 2019.
• Jeffrey Lund, Piper Armstrong, Wilson Fearn, Stephen Cowley, Courtni Byun, Jordan Boyd-Graber, and Kevin Seppi. Automatic and Human Evaluation of Local Topic Quality. Association for Computational Linguistics, 2019. [Code] [Bibtex]
  
  

  @inproceedings{Lund:Armstrong:Fearn:Cowley:Byun:Boyd-Graber:Seppi-2019,
  	Title = {Automatic and Human Evaluation of Local Topic Quality},
  	Author = {Jeffrey Lund and Piper Armstrong and Wilson Fearn and Stephen Cowley and Courtni Byun and Jordan Boyd-Graber and Kevin Seppi},
  	Booktitle = {Association for Computational Linguistics},
  	Year = {2019},
  	Location = {Florence, Italy},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2019_acl_local.pdf},
  }
  

• Varun Kumar, Alison Smith, Leah Findlater, Kevin Seppi, and Jordan Boyd-Graber. Why Didn't You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models. Association for Computational Linguistics, 2019. [Bibtex]
  
  

  @inproceedings{Kumar:Smith:Findlater:Seppi:Boyd-Graber-2019,
  	Author = {Varun Kumar and Alison Smith and Leah Findlater and Kevin Seppi and Jordan Boyd-Graber},
  	Booktitle = {Association for Computational Linguistics},
  	Year = {2019},
  	Location = {Florence, Italy},
  	Title = {Why Didn't You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2019_acl_control.pdf},
  }
  

• Dasha Pruss, Yoshinari Fujinuma, Ashlynn Daughton, Michael Paul, Brad Arnot, Danielle Szafir, and Jordan Boyd-Graber. Zika discourse in the Americas: A multilingual topic analysis of Twitter. PlosOne, 2019. [Data] [Bibtex]
  
  

  @article{Pruss:Fujinuma:Daughton:Paul:Arnot:Szafir:Boyd-Graber-2019,
  	Author = {Dasha Pruss and Yoshinari Fujinuma and Ashlynn Daughton and Michael Paul and Brad Arnot and Danielle Szafir and Jordan Boyd-Graber},
  	Journal = {PlosOne},
  	Year = {2019},
  	Title = {Zika discourse in the Americas: A multilingual topic analysis of {Twitter}},
  	Url = {https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0216922},
  }
  

• Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples. Transactions of the Association for Computational Linguistics, 2019. [Code] [Videos] [Data] [Bibtex]
  
  

  @article{Wallace:Rodriguez:Feng:Yamada:Boyd-Graber-2019,
  	Title = {Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples},
  	Author = {Eric Wallace and Pedro Rodriguez and Shi Feng and Ikuya Yamada and Jordan Boyd-Graber},
  	Booktitle = {Transactions of the Association for Computational Linguistics},
  	Year = {2019},
  	Volume = {10},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2019_tacl_trick.pdf},
  }
  

  
  
• Jeffrey Lund, Stephen Cowley, Wilson Fearn, Emily Hales, Kevin Seppi. Labeled Anchors and a Scalable, Transparent, and Interactive Classifier. Empirical Methods in Natural Language Processing. Brussels, Belgium. 2018.
• Alison Smith, Varun Kumar, Jordan Boyd-Graber, Kevin Seppi, and Leah Findlater. User-Centered Design and Evaluation of a Human-in-the-Loop Topic Modeling System. Intelligent User Interfaces, 2018. [Bibtex]
  
  

  @inproceedings{Smith:Kumar:Boyd-Graber:Seppi:Findlater-2018,
  	Author = {Alison Smith and Varun Kumar and Jordan Boyd-Graber and Kevin Seppi and Leah Findlater},
  	Booktitle = {Intelligent User Interfaces},
  	Year = {2018},
  	Title = {User-Centered Design and Evaluation of a Human-in-the-Loop Topic Modeling System},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2018_iui_itm.pdf},
  }
  

  Alison won a best student paper honorable mention (3 out of 300)
• Paul Felt, Eric Ringger, Kevin Seppi, and Jordan Boyd-Graber. Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types. International Conference on Computational Linguistics, 2018. [Bibtex]
  
  

  @inproceedings{Felt:Ringger:Seppi:Boyd-Graber-2018,
  	Title = {Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types},
  	Author = {Paul Felt and Eric Ringger and Kevin Seppi and Jordan Boyd-Graber},
  	Booktitle = {International Conference on Computational Linguistics},
  	Year = {2018},
  	Location = {Santa Fe, New Mexico},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2018_coling_measurements.pdf},
  }
  

• Jeff Lund, Connor Cook, Kevin Seppi, and Jordan Boyd-Graber. Tandem Anchoring: A Multiword Anchor Approach for Interactive Topic Modeling. Association for Computational Linguistics, 2017. [Code] [Bibtex]
  
  

  @inproceedings{Lund:Cook:Seppi:Boyd-Graber-2017,
  	Title = {Tandem Anchoring: A Multiword Anchor Approach for Interactive Topic Modeling},
  	Author = {Jeff Lund and Connor Cook and Kevin Seppi and Jordan Boyd-Graber},
  	Booktitle = {Association for Computational Linguistics},
  	Year = {2017},
  	Location = {Vancouver, British Columbia},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2017_acl_multiword_anchors.pdf},
  }
  

  
  
• Alison Smith, Varun Kumar, Jordan Boyd-Graber, Kevin Seppi, and Leah Findlater. Accounting for Input Uncertainty in Human-in-the-Loop Systems. CHI 2017 Designing for Uncertainty Workshop, 2017. [Bibtex]
  
  

  @inproceedings{Smith:Kumar:Boyd-Graber:Seppi:Findlater-2017,
  	Author = {Alison Smith and Varun Kumar and Jordan Boyd-Graber and Kevin Seppi and Leah Findlater},
  	Booktitle = {CHI 2017 Designing for Uncertainty Workshop},
  	Year = {2017},
  	Location = {Denver, CO},
  	Title = {Accounting for Input Uncertainty in Human-in-the-Loop Systems},
  	Url = {http://visualization.ischool.uw.edu/hci_uncertainty/papers/Paper11.pdf},
  }
  

• You Lu, Jeff Lund, and Jordan Boyd-Graber. Why ADAGRAD Fails for Online Topic Modeling. Empirical Methods in Natural Language Processing, 2017. [Bibtex]
  
  

  @inproceedings{Lu:Lund:Boyd-Graber-2017,
  	Title = {Why ADAGRAD Fails for Online Topic Modeling},
  	Author = {You Lu and Jeff Lund and Jordan Boyd-Graber},
  	Booktitle = {Empirical Methods in Natural Language Processing},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2017_emnlp_adagrad_olda.pdf},
  	Year = {2017},
  	Location = {Copenhagen, Denmark},
  }
  

• Tak Yeon Lee, Alison Smith, Kevin Seppi, Niklas Elmqvist, Jordan Boyd-Graber, and Leah Findlater. The Human Touch: How Non-expert Users Perceive, Interpret, and Fix Topic Models. International Journal of Human-Computer Studies, 2017. [Journal] [Bibtex]
  
  

  @article{Lee:Smith:Seppi:Elmqvist:Boyd-Graber:Findlater-2017,
  	Author = {Tak Yeon Lee and Alison Smith and Kevin Seppi and Niklas Elmqvist and Jordan Boyd-Graber and Leah Findlater},
  	Journal = {International Journal of Human-Computer Studies},
  	Year = {2017},
  	Title = {The Human Touch: How Non-expert Users Perceive, Interpret, and Fix Topic Models},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2017_ijhcs_human_touch.pdf},
  }
  

• Jordan Boyd-Graber. Humans and Computers Working Together to Measure Machine Learning Interpretability. The Bridge, 2017. [Journal] [Bibtex]
  
  

  @article{Boyd-Graber-2017,
  	Author = {Jordan Boyd-Graber},
  	Journal = {The Bridge},
  	Year = {2017},
  	Title = {Humans and Computers Working Together to Measure Machine Learning Interpretability},
  	Volume = {47},
  	Pages = {6--10},
  }
  

• Alison Smith, Tak Yeon Lee, Forough Poursabzi-Sangdeh, Jordan Boyd-Graber, Kevin Seppi, Niklas Elmqvist, and Leah Findlater. Evaluating Visual Representations for Topic Understanding and Their Effects on Manually Generated Labels. Transactions of the Association for Computational Linguistics, 2017. [Journal] [Data] [Bibtex]
  
  

  @article{Smith:Lee:Poursabzi-Sangdeh:Boyd-Graber:Seppi:Elmqvist:Findlater-2017,
  	Author = {Alison Smith and Tak Yeon Lee and Forough Poursabzi-Sangdeh and Jordan Boyd-Graber and Kevin Seppi and Niklas Elmqvist and Leah Findlater},
  	Journal = {Transactions of the Association for Computational Linguistics},
  	Year = {2017},
  	Title = {Evaluating Visual Representations for Topic Understanding and Their Effects on Manually Generated Labels},
  	Volume = {5},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2017_tacl_eval_tm_viz.pdf},
  	Pages = {1--15},
  }
  

• Forough Poursabzi-Sangdeh, Jordan Boyd-Graber, Leah Findlater, and Kevin Seppi. ALTO: Active Learning with Topic Overviews for Speeding Label Induction and Document Labeling. Association for Computational Linguistics, 2016. [Code] [Bibtex]
  
  

  @inproceedings{Poursabzi-Sangdeh:Boyd-Graber:Findlater:Seppi-2016,
  	Title = {ALTO: Active Learning with Topic Overviews for Speeding Label Induction and Document Labeling},
  	Author = {Forough Poursabzi-Sangdeh and Jordan Boyd-Graber and Leah Findlater and Kevin Seppi},
  	Booktitle = {Association for Computational Linguistics},
  	Year = {2016},
  	Location = {Berlin, Brandenburg},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2016_acl_doclabel.pdf},
  }
  

• Alison Smith, Tak Yeon Lee, Forough Poursabzi-Sangdeh, Jordan Boyd-Graber, Kevin Seppi, Niklas Elmqvist, and Leah Findlater. Human-Centered and Interactive: Expanding the Impact of Topic Models. CHI Human Centred Machine Learning Workshop, 2016. [Bibtex]
  
  

  @inproceedings{Smith:Lee:Poursabzi-Sangdeh:Boyd-Graber:Seppi:Elmqvist:Findlater-2016,
  	Author = {Alison Smith and Tak Yeon Lee and Forough Poursabzi-Sangdeh and Jordan Boyd-Graber and Kevin Seppi and Niklas Elmqvist and Leah Findlater},
  	Booktitle = {CHI Human Centred Machine Learning Workshop},
  	Year = {2016},
  	Location = {San Jose, CA},
  	Title = {Human-Centered and Interactive: Expanding the Impact of Topic Models},
  }
  

  
  
• Md Arafat Sultan, Jordan Boyd-Graber, and Tamara Sumner. Bayesian Supervised Domain Adaptation for Short Text Similarity. North American Association for Computational Linguistics, 2016. [Talk] [Bibtex]
  
  

  @inproceedings{Sultan:Boyd-Graber:Sumner-2016,
  	Title = {Bayesian Supervised Domain Adaptation for Short Text Similarity},
  	Author = {Md Arafat Sultan and Jordan Boyd-Graber and Tamara Sumner},
  	Booktitle = {North American Association for Computational Linguistics},
  	Year = {2016},
  	Location = {San Diego, CA},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2016_naacl_sts.pdf},
  }
  

  
  
• Paul Felt, Eric Ringger, and Kevin Seppi. Semantic Annotation Aggregation with Conditional Crowdsourcing Models and Word Embeddings. International Conference on Computational Linguistics. Osaka, Japan. 2016.
• Jeff Lund and Chace Ashcraft, Andrew McNabb, and Kevin Seppi. Mrs: High Performance MapReduce for Iterative and Asynchronous Algorithms in Python. Workshop on Python for High-Performance and Scientific Computing. Salt Lake City, Utah. 2016.
• Viet-An Nguyen, Jordan Boyd-Graber, Philip Resnik, and Kristina Miler. Tea Party in the House: A Hierarchical Ideal Point Topic Model and Its Application to Republican Legislators in the 112th Congress. Association for Computational Linguistics, 2015. [Talk] [Code] [LaTeX] [Bibtex]
  
  

  @inproceedings{Nguyen:Boyd-Graber:Resnik:Miler-2015,
  	Title = {Tea Party in the House: A Hierarchical Ideal Point Topic Model and Its Application to Republican Legislators in the 112th Congress},
  	Author = {Viet-An Nguyen and Jordan Boyd-Graber and Philip Resnik and Kristina Miler},
  	Booktitle = {Association for Computational Linguistics},
  	Year = {2015},
  	Location = {Beijing, China},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2015_acl_teaparty.pdf},
  }
  

  
  

  Accessible Abstract: In the mid 2010s, the Republican party in the United States diverged: mainstream conservatives split from the so-called "tea party" caucus. However, the primary statistical tool for analyzing political factions in legislative bodies (ideal point models) fail to account for these changes. This is because the schism is not fully reflected in voting patterns but rather in how politicians present themselves: thus we need to extend these models to capture not just how politicians vote but also how they frame particular issues. This paper proposes a new model to capture framing differences within a voting block to start explaining the new subcoalitions of the republican caucus.

• Paul Felt, Eric Ringger, Jordan Boyd-Graber, and Kevin Seppi. Making the Most of Crowdsourced Document Annotations: Confused Supervised LDA. Conference on Computational Natural Language Learning, 2015. [Talk] [Bibtex]
  
  

  @inproceedings{Felt:Ringger:Boyd-Graber:Seppi-2015,
  	Title = {Making the Most of Crowdsourced Document Annotations: Confused Supervised {LDA}},
  	Author = {Paul Felt and Eric Ringger and Jordan Boyd-Graber and Kevin Seppi},
  	Booktitle = {Conference on Computational Natural Language Learning},
  	Year = {2015},
  	Location = {Beijing, China},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2015_conll_cslda.pdf},
  }
  

  This paper received the best paper award at CoNLL
  
• Yi Yang, Doug Downey, and Jordan Boyd-Graber. Efficient Methods for Incorporating Knowledge into Topic Models. Empirical Methods in Natural Language Processing, 2015. [Code] [Bibtex]
  
  

  @inproceedings{Yang:Downey:Boyd-Graber-2015,
  	Title = {Efficient Methods for Incorporating Knowledge into Topic Models},
  	Author = {Yi Yang and Doug Downey and Jordan Boyd-Graber},
  	Booktitle = {Empirical Methods in Natural Language Processing},
  	Year = {2015},
  	Location = {Lisbon, Portugal},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2015_emnlp_fast_priors.pdf},
  }
  

• Stephen H. Bach, Bert Huang, Jordan Boyd-Graber, and Lise Getoor. Paired-Dual Learning for Fast Training of Latent Variable Hinge-Loss MRFs. International Conference on Machine Learning, 2015. [Video] [Bibtex]
  
  

  @inproceedings{Bach:Huang:Boyd-Graber:Getoor-2015,
  	Title = {Paired-Dual Learning for Fast Training of Latent Variable Hinge-Loss MRFs},
  	Author = {Stephen H. Bach and Bert Huang and Jordan Boyd-Graber and Lise Getoor},
  	Location = {Lille, France},
  	Booktitle = {International Conference on Machine Learning},
  	Year = {2015},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2015_icml_paired_dual.pdf},
  }
  

• Thang Nguyen, Jordan Boyd-Graber, Jeff Lund, Kevin Seppi, and Eric Ringger. Is your anchor going up or down? Fast and accurate supervised topic models. North American Association for Computational Linguistics, 2015. [Bibtex]
  
  

  @inproceedings{Nguyen:Boyd-Graber:Lund:Seppi:Ringger-2015,
  	Title = {Is your anchor going up or down?  {F}ast and accurate supervised topic models},
  	Author = {Thang Nguyen and Jordan Boyd-Graber and Jeff Lund and Kevin Seppi and Eric Ringger},
  	Booktitle = {North American Association for Computational Linguistics},
  	Year = {2015},
  	Location = {Denver, Colorado},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2015_naacl_supervised_anchor.pdf},
  }
  

• Paul Felt, Kevin Black, Eric Ringger, Kevin Seppi, Robbie Haertel. Early Gains Matter: A Case for Preferring Generative over Discriminative Crowdsourcing Models. North American Association for Computational Linguistics. Denver, Colorado. 2015.
• Alison Smith, Jason Chuang, Yuening Hu, Jordan Boyd-Graber, and Leah Findlater. Concurrent Visualization of Relationships between Words and Topics in Topic Models. ACL Workshop on Workshop on Interactive Language Learning, Visualization, and Interfaces, 2014. [Bibtex]
  
  

  @inproceedings{Smith:Chuang:Hu:Boyd-Graber:Findlater-2014,
  	Title = {Concurrent Visualization of Relationships between Words and Topics in Topic Models},
  	Author = {Alison Smith and Jason Chuang and Yuening Hu and Jordan Boyd-Graber and Leah Findlater},
  	Booktitle = {ACL Workshop on Workshop on Interactive Language Learning, Visualization, and Interfaces},
  	Year = {2014},
  	Location = {Baltimore, Maryland},
  }
  

• Yuening Hu, Jordan Boyd-Graber, Brianna Satinoff, and Alison Smith. Interactive Topic Modeling. Machine Learning, 2014. [Journal] [Frontend Code] [Backend Code] [Bibtex]
  
  

  @article{Hu:Boyd-Graber:Satinoff:Smith-2014,
  	Title = {Interactive Topic Modeling},
  	Author = {Yuening Hu and Jordan Boyd-Graber and Brianna Satinoff and Alison Smith},
  	Journal = {Machine Learning},
  	Url = {http://umiacs.umd.edu/~jbg//docs/2014_mlj_itm.pdf},
  	Year = {2014},
  	Volume = {95},
  	Pages = {423--469},
  	Publisher = {Springer},
  }
  

• Paul Felt, Kevin Black, Eric Ringger, Kevin Seppi, Robbie Haertel. On Multinomial vs. Log-linear Crowdsourcing Models with Mean-field Variational Inference. NIPS 2014 Workshop on Crowdsourcing and Machine Learning. Montreal, Canada. 2014.

• Topical Guide
• Interactive Anchor TM
• Active Learning with Topic Models
• Active Learning with Topic Overviews

• Topic Labels and Quality Ratings

• Marcus Smith. Prof. Ringger and Natural Language Processing. Thinking Aloud, 2014. [Bibtex]
  
  

  @article{Smith-2014,
  	Author = {Marcus Smith},
  	Year = {2014},
  	Title = {Prof. Ringger and Natural Language Processing},
  	Journal = {Thinking Aloud},
  	Url = {https://cs.byu.edu/article/prof-ringger-and-natural-language-processing},
  }
  

Acknowledgments

This work is supported by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the researchers and do not necessarily reflect the views of the National Science Foundation.