Image: John Tredennick, Merlin & Midjourney
Since its debut in late 2022, ChatGTP has become the focal point of discussions in almost every field that involves some form of computing. The advent of these new generative large language models (LLMs) heralds a transformative era, comparable to the groundbreaking inventions of the printing press, steam engine, cell phones, and the Internet. It also raises a number of important questions regarding the risks inherent in this new generative AI technology, both now and in the future.
Our focus today is on tasks relating to investigations and ediscovery. We hope to illustrate how modern search platforms can take advantage of a generative AI system like GPT to reengineer discovery workflow, drastically improving its efficiency and cost-effectiveness.
For these exercises we will be using B2, an internal system created in the Merlin lab to explore how we might best integrate the analytical power of GPT in a traditional discovery platform. Our goal is to identify relevant documents in response to a natural language prompt and submit them to GPT for analysis and response.
Image: John Tredennick
B2’s job is to:
- Comprehend the initial prompt and identify relevant documents from the database that could aid in responding to it.
- Distill these documents into a summary that aligns with the informational requirements of the initial prompt.
- Condense this information into a form that can be fed into GPT for response. Depending on the complexity and volume of the information, this may necessitate breaking our efforts into a series of prompts and responses.
- Receive GPT’s response, which should be based on both the reviewed information and the content of the original prompt.
- Record this information, and transmit it back to the user.
These steps are complicated by a feature known as a “context window.” This is akin to GPT’s working memory, which includes the prompt, the text it analyzes and its subsequent response. A software system like B2 must be cognizant of the memory limitations of each LLM it interacts with, tailoring the text it feeds into the LLM to stay within the boundaries of the context window. Limited context windows require that the system hold pieces of information in separate storage until they are needed.
From Search Hits to Discovery Answers
Before we jump into specifics, let us offer this observation. The integration of an LLM like GPT into a discovery platform marks the beginning of a new era for investigations and ediscovery.
Up until now, keywords were the primary means to find relevant documents. The search engine’s job was to locate potential candidates that might be responsive to your information needs. Once this step was complete, the search engine’s job ended. The onus then fell on you and your team to read, analyze, and interpret the results, a process that was often tedious, time-consuming, and expensive.
Generative AI systems like GPT can take discovery beyond simple search. They can analyze result sets and use them to answer questions or otherwise provide meaningful information in response to your prompt.
Thus, for the first time in history, we have at our fingertips a generative AI system that can assist with the second half of the discovery process–ESI review and analysis. The second half is where all the money (and the time) goes.
Keep an eye on how well B2 performs its tasks through these exercises. We believe you’ll be impressed with its analytical prowess and capabilities.
Our Topic for This Exercise
Our discovery collection consists of approximately 300,000 emails that Jeb Bush made public from his two terms as Governor of Florida. NIST used these emails for several of its annual Text Retrieval Conferences (TREC). Among other things, the program coordinators and a team of reviewers went through the emails and created a series of topics to be used for research.
For this exercise, we will focus on this legal track topic:
Slot Machines — All documents concerning the definition, legality, and licensing of “slot machines” in Florida.
We will ask B2 to find relevant documents, summarize and synthesize their content and send them on to GPT for analysis and response.
Let’s start by asking GPT to give us an overview of discussions around our topic.
This is an interesting and well-written synthesis of some of the concerns about slot machines in Florida. GPT not only summarized the points made and the people making them, but provided links to the documents used to develop its answer.
Imagine having something like this handed to you at the beginning of an investigation (or at the start of a case). Prior to generative AI, it might take an attorney-led team hours (or even days) to find, analyze and report out this kind of information. GPT can do it in minutes.
Using GPT to Summarize Documents
During the course of responding to prompts, we ask GPT to summarize each document it finds, providing information about the people involved and communication dates. Summaries aren’t meant to be a substitute for the original documents but they can surely be helpful as a starting point.
Here are several examples that GPT prepared before crafting its response. They will give you an idea of GPT’s capabilities in this regard:
B2’s document summaries strike us as hugely valuable for an investigation or discovery effort. They can be persisted for later viewing or exported as part of a report.
Using GPT to Synthesize Information
Along with summarization, GPT does an excellent job at synthesizing information. In this case we will ask GPT to report on concerns regarding slot machines. This request is similar to our starting request but we will give GPT a bit more room to respond here. It will also give you a chance to see how GPT responds to different types of prompts.
You can quickly see the value of these types of reports.
Using GPT to Identify Related Statutes and Regulations
We have seen references to a number of rules and regulations that bear on these discussions. Let’s ask GPT to summarize them.
This can be helpful as well.
Using GPT to Create an Investigation Report
Our final step is to use GPT to create what we would call an investigation report. Assume that an investigator has found a number of important documents through any of a variety of methods (e.g. keyword searches, witness meetings). Rather than asking a series of questions about the document, the investigator could simply ask for a comprehensive report (designed to meet your needs).
The prompt might ask GPT to include this kind of information:
- Individuals involved, including their roles
- A timeline of discussions
- Summaries of the issues involved
- Rules and regulations involved in the discussion
Take a look at the report we received.
We plan to develop a series of report formats and content options to standardize the process. At any point, you can create your own report format simply by specifying what information you need and how you want to see it. Imagine having these kinds of capabilities at your fingertips.
Conclusion
Welcome to a new era of Discovery–one where we can quickly move from search hits to discovery answers with the extraordinary power of artificial intelligence. By seamlessly integrating an LLM with an algorithmic search engine like Sherlock, legal professionals can harness the immense capabilities of large language models to streamline discovery processes and quickly dive deeper into the key documents that are most relevant for the case.
The applications we have explored in this article merely scratch the surface of an LLM’s potential to assist in investigations and discovery efforts. Using an integrated system like B2 to help in synthesizing information, summarizing documents, answering questions and creating investigation reports, will make investigations and discovery more efficient, improving outcomes and saving time and money.
Looking ahead, we believe that LLMs, when integrated with systems like B2 , will dramatically reduce the number of attorneys required for “first pass review” of large document populations (for productions or return review).[1] If our hypothesis is correct–and there are field reports supporting our conclusions–the impact on investigations and ediscovery could be revolutionary.
[1] We discussed some of our early research on using GPT to augment or replace human review teams in this article: Will ChatGPT Replace Ediscovery Review Teams? (Law.com 02/21/2023). We plan to explore this topic further in subsequent papers discussing how an LLM like GPT could be integrated into a TAR 1.0 or 2.0 process to make that review even more efficient and cost-effective.
