What do you do when you are hit with litigation or a government investigation where you have hundreds of gigabytes, potentially meaning hundreds of thousands of pages of documents, and a short timeline for production? There are numerous technology-assisted ways to deal with large amounts of data, some of which include early cases assessment (ECA) tools, simple culling, and advanced analytics. Even with all of the technology available, large cases will likely require some form of manual review, but there are limited resources available that explain how to approach such a task. As such, what follows here is a step by step guide for how to manage human review, including both managing the human aspect of review and some basic technical considerations to take into account along the way. This should serve as a starting point for someone new to managing the process or who needs ideas for improving their large scale document review.

PREPARING FOR THE REVIEW

Assessing the Size and Content of Your Data Collection

The size and content of your data collection will determine several factors of your review. The first thing you should do to get control of the size and content of your review is to implement some form of ECA or culling. It is paramount to understand what you are up against as early as possible, and to reduce the amount of data that needs to be reviewed as much as possible before beginning your review. The various methods and technologies you could use to assess your data are beyond the scope of this article, but understand that there are many ways to accomplish this, from applying basic date ranges and search terms to advanced analytics.

Once you have a handle on the size and content of what needs to be reviewed, you can look towards planning your review. If you are using a managed review service provider, determine what their team’s speed of review is in order to decide how many reviewers you will need in order to complete the review in time, building in time for Quality Control (QC) review. Also consider, based on the content of the documents, whether you will need any specialty reviewers, such as foreign language reviewers.

Creating and Utilizing Protocols                                                  

It is important to have protocols that your review team can read, understand, and readily access. There are two prongs to document review protocols: substantive and technical, and there may be overlap between the two.

• Substantive or Subject Matter Protocols: In the event you have a data set that necessitates bringing in contract attorneys (who are not yet subject matter experts) to assist with the review, you will need substantive protocols in place to train the reviewers on the subject matter of the project and the case. The purpose of substantive protocols is to take what is in the minds of merit counsel, as well as the applicable information in the case file materials, and put that information and analysis into a form that can be transferred to the review team for training as well as for use throughout the project. For example, as far as file materials, you may want to include the Request for Production items in the protocol. Also, assess the file for any other documents, such as sample hot documents, that may help the reviewers to understand the case and know what to look for. Within the protocols communicate to the review team your theory of the case and what facts help or hurt your case. Also, timelines and a factual overview can be a helpful addition to Substantive Protocols.
• Technical Protocols: These protocols are designed to ensure the review team is trained on how to use the review platform and any other technologies that you are using for the review and to communicate the structure and logic of the tagging/coding layouts that the reviewers will be using to document their decisions. If any reviewers are not familiar with the review platform, provide a training session to bring them up to speed.

Substantive and technical protocols overlap in the coding panel that has been designed for the particular case. It is important to properly plan the types of coding you will need and how they should be set up and organized, as coding panels usually have many options at your disposal. Your coding panel may be as simple as only including options for selecting responsive / non-responsive document tags as well as privilege tags. You may also want to add subject matter related tags, a tag for hot docs, or other tags to escalate documents to higher level reviewers. You can also add an attorney note section where you can type in specific notes about a document. In addition to the tags that you will be using to make your coding decisions, you can include metadata information about the document for easy reference, such as the date or title of the document. There also may be the built in ability to view duplicate documents or family members[1], as well as a history of who has previously tagged a document.

Deciding on an Approach

Decide who will be designated to review the documents: subject matter experts, contract attorneys, or a hybrid team. This decision is often based on the size of your data collection and possibly on what technology you will be using. For a large volume of documents, you may need to bring in contract attorneys for the first pass review and use law firm associates for second level review. For a smaller volume, you may want to use only subject matter experts. Also, decide on how you will handle relevance review versus privilege review. This will also be impacted by the size of your data collection.

If you have a larger scale review with a review group that contains fewer subject matter experts, then it may make sense to compartmentalize and have one group review for relevance only, narrowing down the collection, and then have another group review the relevant documents for privilege. However, beware with this approach. If you are only reviewing relevant documents for privilege, you need to decide in advance how you are going to treat “families^[2]” of documents. Industry standard in most scenarios is to produce full families of relevant documents. Since you do not want to produce any documents that have not been first reviewed for privilege, the best practice is to include entire families in your privilege review.

For a smaller volume review where subject matter experts are to put eyes on all of the documents, it is most efficient to have the subject matter experts review for both responsiveness and privilege in one pass.

Finally, you may wish to use a hybrid approach for relevance and privilege review and for your decision as to whom you assign documents for review. One example of a hybrid approach is to try to identify privileged documents using prefabricated search terms and have subject matter experts, such as law firm associates, review the documents returned by the search for both responsiveness and privilege, while the contract attorneys review the remaining documents for responsiveness.  The review set assigned to the contract attorneys may then be reviewed for privilege either by those contract attorneys as they review for responsiveness, or in a separate review for privilege only by law firm associates, depending on what makes sense for your particular project.

CONDUCTING THE REVIEW

Managing the Review and the Review Team

Do not take all of the prep work steps above and then put your review on auto pilot. Ensure that your review is dynamic, not static. In order to accomplish a dynamic, living review, you will need a system in place that allows for change and adjustment as the process goes along. Do not think you can “set it and forget it.” The more you learn about your data, the more you will need to adjust your process. The most important aspect of this is having open lines of communication between the decision-makers and the people who are actually laying eyes on the documents. This communication needs to operate in both directions: contract attorneys asking questions and decision-makers giving feedback.

This is where effective management of people becomes crucial to document review. It is important to treat your reviewers as a valuable part of the process, with a stake in the outcome rather than as “hired guns.” Be sure to give them consistent, concrete feedback on the job they are doing. You want the contract attorneys to feel comfortable contacting their managers to ask questions or inform them of problems and issues they have encountered as they review. You will need a method for communicating changes and giving feedback to reviewers in an atmosphere that makes decision-makers approachable. One method is to create a project notebook kept in the review room (or its equivalent electronic form kept in a virtual meeting room) and give access to all reviewers, who are then notified when changes are made. Certain review platforms have questions tabs where reviewers can leave questions and decision-makers can answer, for all to see, thus keeping everyone on the same page.

Implementing Quality Control (QC)

QC should be on-going throughout the project. Some key pointers for successful QC are outlined below.

• Use statistical sampling[3] to identify potential issues early on and determine whether each issue identified is a systemic problem or an issue isolated to a certain reviewer (i.e.: does the entire group need more training, or should managers focus on certain reviewers?).
• Keep metrics. It is a good idea to share the metrics with the reviewers. Are they reviewing too slow or too fast? Talk to your reviewers. If they are moving fast, but have a high overturn rate, discuss this: “We need you to slow down and make sure you are making good calls. Do you need more training?” If they are slow but accurate, tell them: “Your quality is good, but we need you to move faster. You should have confidence to know your calls are correct, don’t second guess yourself, go ahead and get your pace up.”
• Treat your contract attorneys as more than drones who read for relevance. This will serve the purpose of making them feel like they are contributing so that they want to work on the project. This gives you more eyes with a stake in the game, and reviewers who are looking at the data using all of the legal skills at their disposal.

Creating a Privilege Log

In all aspects of eDiscovery it is best to begin with the end in mind. This also goes for creating a privilege log. First, you will need to know what fields the log will contain based on your agreement with opposing counsel. Most of the informational fields required for your log can probably be exported straight out of the review software using metadata. For example, if the log is supposed to include the sent date and subject of emails, you can export that straight from the metadata.

Next comes what might be the most important tip from this article: Have the review team keep the privilege log as they go, not at the end of the review. Decide what the privilege tags will be in your coding panel, based on how you want them to appear in the log. Decide on what your generic “cookie cutter” privilege descriptions will be, and create a text field for privilege log notes in order to appropriately describe privileged documents. This allows this work to be done as the documents are reviewed, rather than at the end of the project. Make sure it is clear in training that the notes in the privilege log text field will be going to the other side. You may then export the chosen fields as an Excel spreadsheet and your privilege log is complete at the same time the review is complete. Be certain to QC it, of course!

PRODUCTION

Decide in advance whether you are going to do a rolling production or if you will review all the documents and produce everything at once. With a large production, most machine time is usually spent imaging the documents to prepare them to be bates stamped and produced. You can do rolling imaging where you periodically image documents that you know will be produced, rather than waiting to image everything when the review is complete. If you end up changing your mind and image something that you later decide not to produce, it is no issue to simply exclude it from the production. Be aware that you may end up having to pay for the imaging of the document and storage of the image regardless of whether it is produced, however.

CONCLUSION

The eDiscovery industry is full of tech savvy professionals looking for creative and innovative methods of utilizing technology to improve review. This includes the technology professionals who invent and build these software tools, which were created to improve and supplement review, not replace it entirely. Human review is necessary, even if only at the highest level (meaning with merit counsel), because it encompasses the entire point of litigation: knowing your facts and arguing the law based on the facts of your case in order to resolve the dispute before the court. Because of the explosion of data that is relevant to litigation and government investigations, supplemental human reviewers are often necessary in addition to merit counsel, and therefore it is good practice to have a system in place for implementing best practices in managing human review when a project calls for it.

[1] You may be able to view parent/child (e.g. email/attachment) relationships in the coding panel in order to hop to documents in the same family.

[2] “Document Family: All parts of a group of documents that are connected to each other for purposes of communication; e.g. an email and its attachments” See http://cdslegal.com/wp-content/uploads/2012/07/52-ediscovery-terms-you-should-know.pdf

[3] “Statistical sampling is the solution. Sampling starts with a defined ESI collection and attempts to draw reasonable inferences about the general collection from an appropriately sized sample of the collection.” William Hamilton, E-DISCOVERY: Sampling Helps Keep Relevance Relevant, available  at http://www.americanbar.org/publications/litigation_journal/2010_11/summer/e-discovery_sampling_helps_keep_relevance_relevant.html

By: Suzanne H. Clark, Esq., CEDS and Parke P. McManis, Esq., RCA, CEDS, Managing Director, Mid-Atlantic, Complete Discovery Source, Inc. 

This post was first published on September 22, 2016, as an article written by Ms. Clark and Mr. McManis on behalf of the Electronic Discovery and Digital Evidence Committee, Business Law Section of the Florida Bar, for the Florida Bar Practice Resource Institute (PRI). The original article can be viewed here.