LBSC 796/INFM 718R
Information Retrieval Systems
Fall 2007
Batch Evaluation Design

The goal of this project requirement is to produce a plan for conducting a batch evaluation (using frozen relevance judgments).

Batch evaluations are designed to be conducted fully automatically. They include at least the following:

The evaluation design needs to balance four desirable characteristics: In your evaluation design, you will need to specify each of the four components of the first group in a way that (in aggregrate) reasonably balances the four desirable characteristics in the second group.

In general, you can do this either by adopting someone else's evaluation design and/or evaluation resources or by creating your own. For example, there are existing test collections for expert finding in both the TREC Enterprise Track and at the University of Amsterdam (see http://staff.science.uva.nl/~kbalog/). The TREC Enterprise Track also has a mailing list search task that resembles in some ways the state department cable search task. In some cases, you may want to draw inspiration from what they did, in other cases you may want to go beyond that and actually use an existing test collection rather than creating your own.

The best way to see what an evaluation design looks like is to read a TREC, CLEF, or NTCIR track overview paper. For example, here are two that I have written recently (and thus that I am intimitely familiar with the details of):

Your plan need not be as detailed a these, of course, because these were written AFTER the evaluation. To see what we had before the evaluation, look at: Of course, you won't need to specify all the submission format issues that we did (since you will be submitting to yourself, these don't need to be standardized in advance), and you will probably choose to evaluate your system for only one task. So 3 or 4 pages should probably suffice for what you will write up.

One think you might want to think about is how you plan to divide your evaluation resources to support both formative and summative evaluation. You need some evaluation data to support development, but testing on your training set is a cardinal sin. So you'll want to divide your available data in some way to allow you to later demonstate your (hopefully) excellent results on a previously unseen part of the test collection.

This assignment will be graded, but (as with all the pieces) the overall project grade will be assigned holistically rather than being determined by a fixed formula.

Doug Oard
Last modified: Oct 22 2007