A Comparative Study of the Effect of Search Feature Design on User Experience in Digital Libraries (DLs)
Yuelin Li, Xiangmin Zhang, Ying Zhang, Jingjing Liu
SCILS, Rutgers University 4 Huntington Street, New Brunswick, NJ 08901, USA

{lynnlee, xzhang, yzhang, jingjing}@scils.rutgers.edu ABSTRACT
This study investigates the impact of different search feature designs in DLs on user search experience. The results indicate that the impact is significant in terms of the number of queries issued, search steps, zero-hits pages returned, and search errors.

3. RELATED WORK
In order to inform interface design in IR systems, Shneiderman and Plaisant [5] clarify searching and propose a five-phase userinterface framework for text searches. Kengeri, et al. [3] compare the four DLs: ACM, IEEE-CS, NCSTRL, and NDLTD and detect some usability problems of these DLs, including confused link of "browse" and "search" in NDLTD; too many constraints in simple search and advanced search, and so on. Another case study [2] on usability inspection of DLs calls for reconsidering the interaction design of search and browsing functions. Shiri and Molberg [6] investigate the interfaces of 33 digital collections but without report of user experience. Rao et al. [4] present different interface designs supporting searching or browsing or both for DLs. Belkin et al. [1] demonstrate that the box mode and asking searchers to describe their information problems at length result in longer queries. However, few studies investigate how existing search feature designs in DLs affect users' experience.

Categories and Subject Descriptors
H.3.7. [Digital libraries] User issues

General Terms
Design, Experimentation

Keywords
search feature design, digital libraries, user search experience

1. INTRODUCTION
This study compares different interface designs for search features in DLs and their impacts on user searching experience. Three DLs tested in this study are the ACM digital library (ACM), the IEEE Xplore (Xplore) digital library, and the IEEE computer society (IEEE CS) digital library. User search experience is operationalized as queries users issued, their search time, search steps, user errors, user satisfaction, and so on. This paper reports the results of our evaluation on the search feature designs in the three DLs.

4. METHOD
Participants: Thirty-five engineering and MLIS students from Rutgers University participated in the study. Task: One search task to locate relevant documents on protecting the on-line repository from fraudulent activity by watermarking was assigned to the participants for searching the three DLs. They were asked to save the first ten results from the most satisfactory result set, and then to identify the relevant and partially relevant items. Experimental design: To avoid the learning effects, a Latin-square design was employed to randomize the order of the three systems tested in the experiment. Procedure: The experiments were conducted in a usability lab. An entry questionnaire and a pre-search questionnaire were administered before the experiment. A post-search questionnaire was filled out after the search, and an exit interview after the experiment was conducted. Think-aloud was required. The whole session of the experiment was recorded.

2. DESIGN CHARACTERISTICS IN THE THREE DIGITAL LIBRARIES
All three DLs cover computer science and computer engineering. Xplore also covers other subjects in engineering. IEEE CS and Xplore have overlaps, e.g, part of their proceedings. All three DLs support both fielded search and full-text search and provide basic and advanced search levels. However, they have different design features for the search function. ACM supports both fielded and full text search as the default mode. IEEE CS also supports both fielded search and full text search. However, it provides only four fields. Full text search is also the default search mode. Although Xplore supports both fielded search and full-text search, the fielded search is set as the default mode for both the basic and advanced search modes, and only in advanced search mode the user can choose full-text search. In terms of results display, ACM presents citation and abstract. IEEE CS presents citation and a few words of the abstract. However, Xplore displays only the citation on the initial result page. We intend to examine how these differences affect users' search experience.
Copyright is held by the author/owner(s). SIGIR'06, August 6­11, 2006, Seattle, Washington, USA. ACM 1-59593-369-7/06/0008.

5. RESULTS
Over 94% of participants rated themselves above the medium level of computer experience, and nearly 50% considered themselves experts in using computers. About 83% considered themselves very experienced with searching on the internet. The search performance of the three systems was compared based on the Average Precision (AP) of the participants' saved results. One-way ANOVA test showed no statistical significance between the three systems in terms of this measure.

669


5.1 User searching experience
User searching experience was measured using the following objective metrics: 1) Average query length: The mean number of words in all queries issued. 2) The number of queries issued. 3) Amount of search time. 4) The number of search steps: It refers to how many steps the subjects move before they finally get the results. 5) The number of zero-hits ("no results") pages returned. 6) The number of user errors related to searching. Table 1: Comparison of ACM, IEEE CS, and Xplore Average query length Mean # of queries issued Mean search time (Second) Mean # of search steps Mean # of "zero-hits" pages Total # of search errors** ACM 3.62 3.26 274.31 11.60 0.69 3 IEEE CS 3.32 2.61 240.92 12.03 0.50 7 Xplore 2.71 6.14 327.91 22.97 3.26 17 F 2.661 9.025* 1.067 7.167* 16.892*

Table 2: User satisfaction ratings on the search function Statement Easy to get start (searching) Made great effort to accomplish the task Satisfaction with final search results Satisfaction with overall ease of search Satisfaction with overall search feature User mean ratings ACM 5.46 4.09 4.77 5.37 5.06 IEEE CS 5.66 3.94 4.77 5.26 4.89 Xplore 4.97 4.4 9 4.71 4.83 4.97

6. DISCUSSION AND CONCLUSION
Our results indicate that given similar users' familiarity with information searching, different system's default search methods ("All field" in Xplore; "All information (including full-text)" in ACM, and "Full-text" in IEEE CS) and display formats of search result may lead to significant differences in the number of query issued, mean number of search steps and zero-hits pages returned, as well as the users' satisfaction ratings on search features. However, this paper only reports part of user searching experience by examining some objective measures and user satisfaction ratings. In the next step of data analysis, how the participants used different search features and which feature resulted in user frustrations will be analyzed. Only after that can a holistic picture of how search interface design impacts user experience be presented.

*Significant difference was found at p<.01, tested by ANOVA ** Significant difference was found at p<.01, tested by Chi-square Table 1 shows that the queries issued to Xplore were relatively shorter than the other two systems, but not significantly. This means that the subjects issued similar lengthy queries to these three systems. Comparing to ACM and IEEE CS, the participants had to issue significantly more search queries in Xplore (F(2, 103)=9.025, p<.01). The subjects spent much more time when searching Xplore, though the differences are not statistically significant. Xplore also required the participants to move significantly more steps than ACM and IEEE CS did for completing the search task (F(2, 103) = 7.167, p<.01). These results indicate that the subjects had to exert more effort to accomplish the search task in Xplore, which may be related to the issue of zero-hits pages. On average, the participants obtained significantly more zero-hits pages from Xplore (F(2, 103) = 16.892, p<.01). This may be one of the reasons that the subjects had to spend more time, move more steps, and issue more queries in Xplore. In terms of user errors, a Chi-square test found a significant difference in users' total number of search errors (2 (2, N=27) = 11.56, p < .01). The participants made most errors with the Xplore and least with ACM.

7. ACKNOWLEDGMENTS
Our thanks to IEEE, Inc. for sponsoring this project.

8. REFERENCES
[1] Belkin, N. J., Cool, C., Kelly, D., Kim, G., Kim, J.-Y., Lee, H.-J., Muresan, G., Tang, M.-C., and Yuan, X.-J. Query length in interactive information retrieval. Proceedings of the 26th annual international ACM SIGIR (Toronto, CA, Jul. 28Aug. 01, 2003) [2] Hartson, H. R., Shivakumar, P., and Perez-Quinones, M. A. Usability inspection of digital libraries: a case study. International Journal of Digital Library, 4 (2004), 108-123. [3] Kengeri, R., Seals, C. D., Harley, H. D., Reddy, H. R., & Fox, E. A. Usability study of digital libraries: ACM, IEEECS, NCSTRL, NDLTD. International Journal of Digital Library, 2 (1999), 157-169. [4] Rao, R., Pedersen, J. O., Hearst, M. A., Mackinlay, J. D., Card, S. K., Masinter, L., Halvorsen, P.-K., & Robertson, G. G. Rich interaction in the digital library. Communications of the ACM, 38, 4 (1995), 29-39. [5] Shenierderman, B., and Plainsant, C. Designing the user interface. Pearson, New York, NY, 2005. [6] Shiri, A. and Molberg, K. Interfaces to knowledge organization systems in Canadian digital library collections. Online information Review, 29, 6 (Nov. 2005), 604-620.

5.2 User satisfaction
Table 2 shows users' satisfaction ratings on the three systems' search features. The ratings were based on 7-point Likert scales. Except the statement of "Made great effort to accomplish the task," for other statements, the higher the rating, the better the feature. There were no statistical significances among the ratings for the three systems. However, the ratings for Xplore were lower than ACM and IEEE CS in most cases. Apparently, many zero hits returned and poor system support and feedback for query construction and refinement in the search feature design affected the participants' satisfaction with Xplore.

670