INST 734
Information Retrieval Systems
Fall 2014
Project User Evaluation Design (Assignment P6)

This document applies only to instructor-designed projects.

The goal for this part of the project is to design the user study that will represent one of the two major parts of your project (the other will be the batch evaluation you will conduct).

You should think of your batch evaluation and your user study as ways of answering different aspects of the same question: "How well does our system help the user achieve their goals?" Because user studies are generally more expensive than batch evaluations (i.e., they take more time to plan, execute and analyze if done well), it is prudent to focus your user study on aspects of the question that are not amenable to batch evaluation. Some examples of these kinds of questions are:

Those are just examples -- you can surely think of many more. Which brings us to the second important point: user studies must be focused if they are to be useful. In other words, you need to decide on a few questions you most want to answer. Think back to exercise E1 and the issues that came to your mind as you tried a new system for a new task. Thinking like a user is one important key to asking the right questions.

Once you have decided what you want to know, you are ready to choose a study design. There are two basic kinds of study designs:

Generally, observational studies are necessary if you want to see things that can only be seen when people are doing an internally-motivated task (because the protocol in a quantitative study must standardize what people search for in order to make it possible to compare performance under different circumstances). Quantitative studies are normally used to compare two variants of your system under controlled conditions (e.g., simple and advanced search interfaces).

Once you have chosen goals for your study and the kind of study design you want to use, you should consult one or more examples of studies using that kind of design. Here's one of each that you could start with:

Of course, your study shouldn't be as ambitious as either of these. You'll probably want to recruit study participants from the members of the class, so you should probably limit the size of your study to four users and the length of your sessions to two hours (including training time, which you shouldn't scrimp on if you hope to learn anything from the study).

Once you have a study design, you should test it on someone who did not contribute to your study design and who will not be one of your actual study participants. This is called a "pilot study", and you will almost surely learn of the need for some improvements (e.g., in how you do training, or in how you collect data). You can go do this with almost anyone since you are not studying them, you are simply trying out your study design. But it is best if you don't use a member of your team (because they know too much!).

Normally you would want to recruit study participants who are representative of the people who would really use a system like the one you are building, and if you can find such people and motivate them to participate that would be great (but see below on IRB requirements if you want to make a publishable study out of this). But in practice, most of you will actually recruit your classmates. The only restriction on this is that if two teams are working on the same problem they should not recruit from each other.

With that as background, here is what should be in your plan:

As general guidelines, your study design should fit comfortably on four pages, and your study (including all planning, the pilot study, the analysis, and reporting results) should require around 50% of the total time available for your project and analysis (8 weeks * 5 hours per week * the number of people on your team * 50% ... for a 2 person team this number is around 40 hours).

If you plan to publish the results of your study, advance approval from the University of Maryland Institutional Review Board (IRB) is required. If your results will not be published, that step is not required. IRB approval can take a month, so get started on this way earlier than the assignment due date if you do want to be able to publish your results.

Submit your user evaluation design using ELMS.

This assignment will be graded, but (as with all the pieces) the overall project grade will be assigned holistically rather than being determined by a fixed formula.

Doug Oard
Last modified: Thu Oct 9 10:04:48 2014