Assignment adapted from James Allan's CMPSCI 646 course (Fall, 2004) at U. Mass.
The purpose of this exercise is to gain some "hands-on" experience in the process of evaluating information retrieval systems. You will be assessing documents that are retrieved in response to two "topics" (statements of information needs). The two search engines we'll be comparing are Google and Teoma (which no longer exists).
gardening wet soil conditions
What special
considerations must be made when planting a garden in very wet soil
conditions? Are there plants that will not work and/or that will work
particularly well? Are there any special techniques that will help
reduce the amount of moisture? Only pages that deal with gardening
are relevant.
[Google results] [Teoma results]
oil vs. propane furnace
Looking for pages that list
the tradeoffs for using a propane furnace rather than a fuel oil
furnace. The pages should provide comparison and are only relevant if
they talk about both. They can talk about other types of heating
(e.g., electric), provided they talk about oil and propane. Propane is
also called LP or LPG (liquid propane [gas]).
[Google results] [Teoma results]
To ensure that everyone evaluates the same hits, results from each search engine have been cached for you; follow the above links. You will evaluate the relevance of a subset of these hits. If your social security number ends in an even digit, evaluate the first 15 hits of each query. If you social security number ends in an odd digit, evaluate the last 15 hits of each query.
Use this Excel spreadsheet to keep track of your relevance judgments. In the column marked "Relevance", enter "R" if you think the document is relevant. Enter "N" if you think the document is not relevant. Add the spreadsheet to your homework Web page. Change the name of the spreadsheet to your last name so that we don't wind up with a dozen files with the same filename.