LBSC 671 - Creating Information Infrastructures
Spring 2014 - Section 0101
Assignment G13 - Machine-Assisted Indexing

The goal of this assignment is to gain some experience with text classification and to get a sense for how well it is working. First, select an online text classification system. You can use any classifier that you wish, but I have tested the following online demos:

• dataTXT-CL Topic Classification
• NLTK Sentiment Classification
• University of Illinois Wikifier (this is two classifiers, one to guess what should be linked, and one to guess where to link it to)
• etcML Twitter Classifier (scroll down to "TRY IT YOURSELF")

First, just try things out a bit to get a sense for what the classifier you have selected is trying to do. Then search the Web to select a dozen or so examples to be classified and (this is important) first classify them yourself. For example, if your classifier guesses whether the text was written by a man or a woman (none of them do, this is just an example), then you would label each text based on YOUR guess. You can base your guess on certain knowledge (because of the way you selected your examples) or you can really just make your best guess (if you really don't know). Then run the classifier on each and compare the classifier's results with your guess (or knowledge) of the right answer. Tabulate the correct and incorrect answers from the classifier (treating your answers as true) and then see if you can explain why the ones that are right are right and why the ones that are wrong are wrong (for example, are they misleading examples? does the classifier pay too much attention to simple features and miss some nuance? etc.). In your answer, tell me:

1. Which classifier you selected and what classification task it was performing. If you use an online demo, please provide the URL.
2. One example of a document that you tried to classify.
3. The accuracy (percent correct) of your classifier.
4. Why you think your classifier made the mistakes it made (if it made any!)