ICAIL 2011 Workshop on
Setting Standards for
Searching Electronically Stored Information
in Discovery Proceedings
(DESI IV Workshop)

June 6, 2011, University of Pittsburgh, Pittsburgh, PA, USA

Purpose | Background | Agenda | Papers | Important Dates
DESI History | Standards | References | Organizing Committee | Archived Materials

Purpose

 The DESI IV workshop is intended to provide a platform for discussion of an open standard governing the elements of a state-of-the-art search for electronic evidence in the context of civil discovery. The dialog at the workshop might take several forms, ranging from a straightforward discussion of how to measure and improve upon the "quality" of existing search processes; to discussing the creation of a national or international recognized standard on what constitutes a "quality process" when undertaking e-discovery searches; to a more ambitious discussion concerning creation of a type of standards "authority" or certification entity that could certify compliance with whatever standard emerges. Issues to be considered include the potential benefits of a standard (e.g., reducing the need for evidentiary hearings on the reasonableness of a process), versus its potential costs (e.g., the risk of inhibiting innovation through early convergence on rigid standards), and timelines.

Background

 The Sedona Conference®, a leading legal think tank, recognized in 2009 that "the legal profession is at a crossroads," in terms of its willingness to embrace automated, analytical and statistical approaches in the area of e-discovery. However, while in the last four years a cottage industry of published case law and commentaries have come into existence recognizing that parties in litigation should reach some kind of agreement on what constitutes a "search protocol," there is no widely agreed-upon set of standards or best practices for how to conduct a reasonable e-discovery search for relevant evidence. The Sedona Conference and others have, however, called out to industry and academia to assist in the further development of standards of what constitutes "best practices" in the area of performing searches of electronic evidence.

In initiating a discussion about standards for what constitutes a "quality" process in e-discovery search, the workshop will serve to achieve the aim of bringing together academia and industry in the development of standards in this area. A recent article on "Evaluation of Information Retrieval in E-Discovery" in the journal Artificial Intelligence and Law's special issue on E-Discovery suggested that:

"One broad class of approaches that has gained currency in recent years ... is known as 'process quality.' Essentially, ... the important thing is that we agree on how each performer of E-discovery services should design measures to gain insight into the quality of the results achieved by their particular process. The design of their process, and of their specific measures, is up to each performer. Of course, some performers might benefit from economies of scale by adopting measures designed by others, but because the measures must fit the process and because process innovation should be not just accommodated but encouraged, forced convergence on specific measures can be counterproductive. So process quality approaches seek to certify the way in which the measurement process is performed rather than what specifically is measured."

Agenda

The full day workshop will be organized in four parts:

Part I: The workshop will begin with an overview of recent developments in e-discovery search, including the needs of the legal profession, recent case law, and recent work on evaluation design.

Part II will consist of a set of presentations selected to illustrate the diversity of current research on e-discovery search processes discussion of some approaches to standard-setting.

Part III will consist of a set of breakout sessions to engage the workshop participants in brainstorming with respect to what process quality standards for e-discovery search might entail. Each breakout group will be asked to initially look at the problem from the perspective of some specific standards-setting process. This discussion will begin with informal discussions over lunch, and then it will be the principal focus of the first session after lunch, ending with brief reports from each group.

Part IV will be a facilitated session with panelists looking back over the day and suggesting ways forward. The session will conclude with a broader discussion of directions for further research, building from two questions:

A final schedule is now available.

Papers

 The Workshop Proceedings is now available. This contains all of the research and position papers in a single PDF file.

Research Papers (peer reviewed)

Position Papers (not peer reviewed)

Important Dates

 The Workshop will begin at 9:00 AM (with registration open from 8:30) and it will end at 5:30 PM.

DESI History

 At the beginning of the second decade of the 21st century, lawyers continue to increasingly face the problem of how to effectively and efficiently conduct searches for relevant documents across increasingly complex, enterprise-wide collections within corporate and institutional settings. Hundreds of millions of documents are now routinely subject to searches across a wide spectrum of litigation and investigatory contexts (e.g., the Lehman Brothers investigation necessitated a review of 350 million pages or 3.5 petabytes of material). In recognition of this situation, in 2006 the United States adopted new rules governing civil litigation in federal courts. These courts have recognized "electronically stored information" (ESI) as a term of art, embracing all forms of electronic documents made subject to the civil discovery process. Under the new rules, opposing parties in federal court litigation now have an early "meet and confer" duty to discuss a range of electronic discovery ("e-discovery") issues, including the continued storage, preservation, and access to ESI in their respective physical and legal custodies.

DESI IV follows three successful prior DESI (Discovery of Electronically Stored Information) Workshops: at ICAIL 2007 (DESI I, Palo Alto) and ICAIL 2009 (DESI III, Barcelona), and an intermediate workshop (DESI II) sponsored by University College London in 2008. In DESI I, a wide array of individuals came together for perhaps the first time to foster engagement between e-discovery practitioners and a broad range of research communities who can contribute to the development of new technologies to support the e-discovery process. The DESI II and III workshops broadened the scope of this discussion to include comparisons of requirements between different national settings and different legal contexts. DESI IV builds on these efforts with its emphasis on standards-setting or benchmarking for e-discovery searches.

Standards

There may well be many good ideas that we can draw on that have been worked out in the context of existing standards-setting processes in other fields, such as:

References

 Much has been published on E-Discovery generally, so no list of references could hope to be complete. Here are a few papers that we know of that we believe would be useful as background reading for the focus of this workshop. Please send recommended additions for this list to oard@umd.edu.
  1. W. Andrews and D. Logan, Early Case Assessment: E-Discovery Beyond Judges and Regulators Is About Risks, Costs and Choices, January 27 (2010).
  2. W. Andrews, D. Logan, J. Bace and S. Childs, E-Discovery SaaS and On-Premises Software Converge at Vendors as They Mature, July 29 (2010).
  3. J. Bace, Communicating Metadata Issues in E-Discovery to the General Counsel, November 10 (2010).
  4. J. Baron, Toward a Federal Benchmarking Standard For Evaluating Information Retrieval Products Used In E-Discovery, The Sedona Conference Journal, 6(1)237-246 (2005)
  5. J. Baron, Law in the Age of Exabytes: Some Further Thoughts on 'Information Inflation' and Current Issues in E-Discovery Search, Richmond Journal of Law and Technology, 17(3), Spring (2011)
  6. M. Grossman, Reflections from the Topic Authorities About the 2009 TREC Legal Track Interactive Task (2010).
  7. M. Grossman and T. Sweeney, What lawyers need to know about search tools, National Law Journal, August (2010).
  8. M. Grossman and G. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, Richmond Journal of Law and Technology, 17(3), Spring (2011)
  9. B. Hedin, S. Tomlinson, J. Baron and D. Oard TREC 2009 Track Overview (2010).
  10. J. Krause, Human-Computer Assisted Search in EDD, Law Technology News, December 20 (2010).
  11. D. Oard, J. Baron, B. Hedin, D. Lewis and S. Tomlinson, Evaluation of Information Retrieval for E-Discovery, Artificial Intelligence and Law 18(4)347-386 (2011).
  12. D. Oard, J. Baron and D. Lewis, Some Lessons Learned to Date from the TREC Legal Track, 2006-2009, February 24 (2010)
  13. H. Roitblat, A. Kershaw and P. Oot., Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, Journal of the American Society for Information Science and Technology, 61(1)70-80 (2010).
  14. The Sedona Conference®, Commentary on Achieving Quality in E-Discovery (2009).

Organizing Committee

 Jason R. Baron, National Archives and Records Administration, USA
Laura Ellsworth, Jones Day, USA
Dave Lewis, David D. Lewis Consulting, USA
Debra Logan, Gartner Research, UK
Douglas W. Oard, University of Maryland, USA

Archived Materials on Submissions

 We invite both e-discovery stakeholders and practitioners from the law, government, and industry, along with researchers on process quality, information retrieval, human language technology, human-computer interaction, artificial intelligence, and other fields connected with e-discovery.

To help craft the program, we encourage the submission of research papers and position papers on emerging best practices in e-discovery search as well as papers discussing the efficacy of standards setting in this area. Accepted position papers and accepted research papers will be made available on the Workshop's Web page and distributed to participants on the day of the event, and some speakers may be selected from among those submitting position papers. See the Call for Submissions for submission details.

Last Update: June 4, 2011