The goal of this task is to determine which documents will be assessed. This task (and the subsequent assessment) is performed by the requesting party.
The requesting party may choose any sampling strategy that he or she believes is likely to yield a reasonable estimate for recall and precision in both the Boolean set and in the set returned by responding party. Approximately 300 documents should be included in the sample (at 1 minute per document, assessment for a sample of this size would take about 5 hours; two weeks are available for assessment).
One possibility would be to use stratified random sampling in a manner similar to the 2008 interactive task. To accomplish this, sort both lists in alphanumeric order, then separate the lists into four sets: (1) returned by both, (2) returned only by Boolean, (3) Returned only by the responding party's best effort, and (4) returned by either. Then RANDOMLY select some reasonable number of samples from each set (e.g., is one set is twice as large as another, it would be reasonable to select twice as many form the same set). Document your selection process carefully, specifically noting how each sampled document was selected. This is crucial if the assessment results are to later be useful as a basis for estimation.
One disadvantage of this approach is that it does not account for the best-first order if one is provided. As an alternative, you may wish to consider the sampling strategy used in the TREC Legal Track ad hoc task, which samples more densely near the top of a ranked list.
On the due date of this assignment (at 6 PM), provide a description of your sampling strategy, the document identifier for each sampled document, and a listing indicating how each sampled document was selected (for use in estimation). Your sampling must be approved by an instructor before you can begin assessment; that will normally take 48 hours, so please plan your assessment schedule accordingly.