LBSC 796/INFM 718R: Information Retrieval Systems | Spring 2011

LBSC 796/INFM 718R: Homework 4

Last Update March 6, 2011 (to specify which judges to use and to correct MAP to AP).

Please turn in this homework by linking it from your course Web page DURING CLASS on the due date. Please do not link it before that time.

Evaluating Systems

This Excel spreadsheet contains seven separate sets of judgments for the hits examined in Homework 2. For one of the topics (you can choose either one), you will:

analyze agreement on the relevance judgments
adjudicate the judgments
use the adjudicated set to evaluate both Bing and Google

1. Agreement on Relevance Judgments

The above spreadsheet contains seven sets of judgments for every document (Web page). The first question you'll answer is: How often do judges agree on relevance? Pick the first (leftmost) three columns of judges. There are four possibilities:

111: all three judges agree that the document is relevant.
110, 101, or 011: two of the three judges agree that the document is relevant.
001, 010, or 100: two of the three judges agree that the document is not relevant.
000: all three judges agree that the document is not relevant.

(note: if you have already done this for all 7 judges, that's fine -- you will just have more possibilities for ways in which they could disagree. Just note in what you turn in that you used all 7 judges if you did that).

For your chosen topic (please say which one you chose!), figure out how often each of those four cases happens (both as counts and as percentages). Turn this information in. Then pick three cases in which the judgments about a particular hit are not all the same (i.e., not 000 or 111), and briefly speculate on why this may be so in these specific cases (i.e., with reference to the contents of the pages), drawing on the nature of relevance as we discussed it in class. Turn this in.

2. Adjudication

Adjudication is simply the process of reconciling inconsistent judgments. Do this by simple majority voting. You do not need to turn anything in for this, but you will need the results for the third question.

3. Evaluation of Bing and Google

Now, evaluate Bing and Google using the adjudicated relevance judgments you just created (for the topic you chose). Make sure you are pooling judgments from both systems! Turn in the following information for both search engines:

Precision at 10, 20, 30
Recall at 10, 20, 30
Uninterpolated Average Precision
A precision-recall graph

In addition, answer the following questions:

Which system is better? Why?
What is the assumption you're making with respect to the completeness of the judgment set when you say that one system is better than another?

Assignment adapted from James Allan's CMPSCI 646 course (Fall 2004) at the University of Massachusetts