M69: Evaluating Search Engines by Modeling the  Relationship Between Relevance and Clicks
Ben Carterette & Rosie Jones
Which ranking of web pages is better for the query "NIPS 2007"?
Is DCG1 > DCG2?

P(DCG1 > DCG2)

Use clicks to model relevance; evaluate probabilistically
Rank   1 Rank   2 Rank   3 Rank   4 Rank   5 Rank   1 Rank   2 Rank   3 Rank   4 Rank   5

Click count 1

Click count 2