Evaluation
LBSC 796/INFM 718R
Session 5, October 7, 2007
Douglas W. Oard
Agenda
•	Questions

•	Evaluation fundamentals

•	System-centered strategies

•	User-centered strategies
IR as an Empirical Discipline
•	Formulate a research question: the hypothesis
•	Design an experiment to answer the question
•	Perform the experiment
–	Compare with a baseline “control”
•	Does the experiment answer the question?
–	Are the results significant? Or is it just luck?
•	Report the results!

Evaluation Criteria
•	Effectiveness
–	System-only, human+system

•	Efficiency
–	Retrieval time, indexing time, index size

•	Usability
–	Learnability, novice use, expert use
IR Effectiveness Evaluation
•	User-centered strategy
–	Given several users, and at least 2 retrieval systems
–	Have each user try the same task on both systems
–	Measure which system works the “best”

•	System-centered strategy
–	Given documents, queries, and relevance judgments
–	Try several variations on the retrieval system
–	Measure which ranks more good docs near the top
Good Measures of Effectiveness
•	Capture some aspect of what the user wants

•	Have predictive value for other situations
–	Different queries, different document collection

•	Easily replicated by other researchers

•	Easily compared
–	Optimally, expressed as a single number
Comparing Alternative Approaches
•	Achieve a meaningful improvement
–	An application-specific judgment call

•	Achieve reliable improvement in unseen cases
–	Can be verified using statistical tests
Evolution of Evaluation
•	Evaluation by inspection of examples
•	Evaluation by demonstration
•	Evaluation by improvised demonstration
•	Evaluation on data using a figure of merit
•	Evaluation on test data
•	Evaluation on common test data
•	Evaluation on common, unseen test data
Which is the Best Rank Order?
Which is the Best Rank Order?
IR Test Collection Design
•	Representative document collection
–	Size, sources, genre, topics, …
•	“Random” sample of representative queries
–	Built somehow from “formalized” topic statements
•	Known binary relevance
–	For each topic-document pair (topic, not query!)
–	Assessed by humans, used only for evaluation
•	Measure of effectiveness
–	Used to compare alternate systems
What is relevance?
Defining “Relevance”
•	Relevance relates a topic and a document
–	Duplicates are equally relevant by definition
–	Constant over time and across users

•	Pertinence relates a task and a document
–	Accounts for quality, complexity, language, …

•	Utility relates a user and a document
–	Accounts for prior knowledge
Another View
Set-Based Effectiveness Measures
•	Precision
–	How much of what was found is relevant?
•	Often of interest, particularly for interactive searching
•	Recall
–	How much of what is relevant was found?
•	Particularly important for law, patents, and medicine
•	Fallout
–	How much of what was irrelevant was rejected?
•	Useful when different size collections are compared
Effectiveness Measures
Single-Figure Set-Based Measures
•	Balanced F-measure
–	Harmonic mean of recall and precision

–	Weakness: What if no relevant documents exist?

•	Cost function
–	Reward relevant retrieved, Penalize non-relevant
•	 For example, 3R+ - 2N+
–	Weakness: Hard to normalize, so hard to average
Single-Valued Ranked Measures
•	Expected search length
–	Average rank of the first relevant document
•	Mean precision at a fixed number of documents
–	Precision at 10 docs is often used for Web search
•	Mean precision at a fixed recall level
–	Adjusts for the total number of relevant docs
•	Mean breakeven point
–	Value at which precision = recall
Automatic Evaluation Model
Ad Hoc Topics
•	In TREC, a statement of information need is called a topic
Questions About the Black Box
•	Example “questions”:
–	Does morphological analysis improve retrieval performance?
–	Does expanding the query with synonyms improve retrieval performance?
•	Corresponding experiments:
–	Build a “stemmed” index and compare against “unstemmed” baseline
–	Expand queries with synonyms and compare against baseline unexpanded queries
Measuring Precision and Recall
Graphing Precision and Recall
•	Plot each (recall, precision) point on a graph
•	Visually represent the precision/recall tradeoff
Uninterpolated Average Precision
–	Average of precision at each retrieved relevant document
–	Relevant documents not retrieved contribute zero to score
Uninterpolated MAP
Visualizing Mean Average Precision
What MAP Hides
Why Mean Average Precision?
•	It is easy to trade between recall and precision
–	Adding related query terms improves recall
•	But naive query expansion techniques kill precision
–	Limiting matches by part-of-speech helps precision
•	But it almost always hurts recall

•	Comparisons should give some weight to both
–	Average precision is a principled way to do this
–	More “central” than other available measures
Systems Ranked by Different Measures
Correlations Between Rankings
Relevant Document Density
Alternative Ways to Get Judgments
•	Exhaustive assessment is usually impractical
–	Topics * documents = a large number!
•	Pooled assessment leverages cooperative evaluation
–	Requires a diverse set of IR systems
•	Search-guided assessment is sometimes viable
–	Iterate between topic research/search/assessment
–	Augment with review, adjudication, reassessment
•	Known-item judgments have the lowest cost
–	Tailor queries to retrieve a single known document
–	Useful as a first cut to see if a new technique is viable
Obtaining Relevance Judgments
•	Exhaustive assessment can be too expensive
–	TREC has 50 queries for >1 million docs each year

•	Random sampling won’t work
–	If relevant docs are rare, none may be found!

•	IR systems can help focus the sample
–	Each system finds some relevant documents
–	Different systems find different relevant documents
–	Together, enough systems will find most of them
Pooled Assessment Methodology
•	Systems submit top 1000 documents per topic

•	Top 100 documents for each are judged
–	Single pool, without duplicates, arbitrary order
–	Judged by the person that wrote the query

•	Treat unevaluated documents as not relevant

•	Compute MAP down to 1000 documents
–	Treat precision for complete misses as 0.0
Does pooling work?
•	Judgments can’t possibly be exhaustive!

•	This is only one person’s opinion about relevance

•	What about hits 101 to 1000?

•	We can’t possibly use judgments to evaluate a system that didn’t participate in the evaluation!
Lessons From TREC
•	Incomplete judgments are useful
–	If sample is unbiased with respect to systems tested 

•	Different relevance judgments change absolute score
–	But rarely change comparative advantages when averaged

•	Evaluation technology is predictive
–	Results transfer to operational settings
Effects of Incomplete Judgments
•	Additional relevant documents are:
–	roughly uniform across systems
–	highly skewed across topics

•	Systems that don’t contribute to pool get comparable results
Inter-Judge Agreement
Effect of Different Judgments
Net Effect of Different Judges
•	Mean Kendall t between system rankings produced from different qrel sets: .938
•	Similar results held for
–	Different query sets
–	Different evaluation measures
–	Different assessor types
–	Single opinion vs. group opinion judgments
Statistical Significance Tests
•	How sure can you be that an observed difference doesn’t simply result from the particular queries you chose?
Statistical Significance Testing
How Much is Enough?
•	Measuring improvement
–	Achieve a meaningful improvement
•	Guideline: 0.05 is noticeable, 0.1 makes a difference
–	Achieve reliable improvement on “typical” queries
•	Wilcoxon signed rank test for paired samples

•	Know when to stop!
–	Inter-assessor agreement limits max precision
•	Using one judge to assess the other yields about 0.8
Recap: Automatic Evaluation
•	Evaluation measures focus on relevance
–	Users also want utility and understandability

•	Goal is to compare systems
–	Values may vary, but relative differences are stable

•	Mean values obscure important phenomena
–	Augment with failure analysis/significance tests

Automatic Evaluation
User Studies
User Studies
•	Goal is to account for interface issues
–	By studying the interface component
–	By studying the complete system

•	Formative evaluation
–	Provide a basis for system development

•	Summative evaluation
–	Designed to assess performance
Questions That Involve Users
•	Example “questions”:
–	Does keyword highlighting help users evaluate document relevance?
–	Is letting users weight search terms a good idea?
•	Corresponding experiments:
–	Build two different interfaces, one with keyword highlighting, one without; run a user study
–	Build two different interfaces, one with term weighting functionality, and one without; run a user study

Blair and Maron (1985)
•	A classic study of retrieval effectiveness	
–	Earlier studies used unrealistically small collections
•	Studied an archive of documents for a lawsuit
–	40,000 documents, ~350,000 pages of text
–	40 different queries
–	Used IBM’s STAIRS full-text system
•	Approach:
–	Lawyers wanted at least 75% of all relevant documents
–	Precision and recall evaluated only after the lawyers were satisfied with the results
Blair and Maron’s Results
•	Mean precision: 79%
•	Mean recall: 20% (!!)
•	Why recall was low?
–	Users can’t anticipate terms used in relevant documents

–	Differing technical terminology
–	Slang, misspellings
•	Other findings:
–	Searches by both lawyers had similar performance
–	Lawyer’s recall was not much different from paralegal’s
Quantitative User Studies
•	Select independent variable(s)
–	e.g., what info to display in selection interface
•	Select dependent variable(s)
–	e.g., time to find a known relevant document
•	Run subjects in different orders
–	Average out learning and fatigue effects
•	Compute statistical significance
–	Null hypothesis: independent variable has no effect
–	Rejected if p<0.05
Variation in Automatic Measures
•	System
–	What we seek to measure
•	Topic
–	Sample topic space, compute expected value
•	Topic+System
–	Pair by topic and compute statistical significance
•	Collection
–	Repeat the experiment using several collections
Additional Effects in User Studies
•	Learning
–	Vary topic presentation order

•	Fatigue
–	Vary system presentation order

•	Topic+User (Expertise)
–	Ask about prior knowledge of each topic
Presentation Order
Batch vs. User Evaluations
•	Do batch (black box) and user evaluations give the same results? If not, why?
•	Two different tasks:
–	Instance recall (6 topics)

–	Question answering (8 topics)
Results
•	Compared of two systems:
–	a baseline system
–	an improved system that was provably better in batch evaluations
•	Results:
Example User Study Results
Qualitative User Studies
•	Observe user behavior
–	Instrumented software, eye trackers, etc.
–	Face and keyboard cameras
–	Think-aloud protocols
–	Interviews and focus groups
•	Organize the data
–	For example, group it into overlapping categories
•	Look for patterns and themes
•	Develop a “grounded theory”

Topicality
Questionnaires
•	Demographic data
–	For example, computer experience
–	Basis for interpreting results

•	Subjective self-assessment
–	Which did they think was more effective?
–	Often at variance with objective results!

•	Preference
–	Which interface did they prefer?  Why?
Summary
•	Qualitative user studies suggest what to build

•	Design decomposes task into components

•	Automated evaluation helps to refine components

•	Quantitative user studies show how well it works
One Minute Paper
•	If I demonstrated a new retrieval technique that achieved a statistically significant improvement in average precision on the TREC collection, what would be the most serious limitation to consider when interpreting that result?