Innovation in the legal industry is particularly challenging because the expectation of accuracy is simply higher than in most sectors. “Move fast and break things” is not an option. Even so, in a period where virtually every large business is investigating the potential of generative AI (gen AI) and large language models (LLMs) to improve efficiency and enable innovation, the legal industry must do the same.
In one instructive comparison, we tested Cornerstone Research’s traditional workflow for document analysis against an LLM-assisted alternative. The corpus of material analyzed comprised close to a hundred thousand publicly filed corporate proxy statements. The LLM-assisted approach performed comparably to human review alone in identifying “true positive” hits, correctly identifying 95%-99% of the examples found via the traditional approach. As compared to traditional keyword search, the LLM-assisted approach was substantially more effective in filtering out “false positive” hits, reducing the pool of documents for which manual review was required by 85%. This enabled the team to broaden their search terms and generate an initial document count that would normally surpass the feasible constraints of human review. The result was a more than 67% increase in the number of relevant examples found.
A second case study assessed LLM capabilities for comprehensive document review and coding, compared against the original team’s manual results. The LLM’s accuracy rate was between 82% and 96%. Although an iterative manual review process remains essential to refining questions and ensuring quality, these findings indicate that after the initial manual review, an LLM pass improves the consistency of results obtained.
Along with the benefits LLMs present, there are some challenges to navigate. LLMs can have an inherent degree of randomness that works at cross-purposes with the need for reproducible results. While teams with the right training and methods can combat this issue, ensuring reproducibility may require a tradeoff of some efficiency. The proper balance must be struck.
In addition, while off-the-shelf tools can be powerful, they can be insufficient for legal clients’ needs in many cases. There are a number of benefits to developing the infrastructure and software to deploy cutting-edge, on-premises LLM models and applications in addition to frontier, third-party models and tools. This approach is well-suited to support the scale, precision, and reproducibility, as well as confidentiality and security, required for work in the legal field.
The remarkably rapid development of LLM technology is exciting, but with new models being released on a near-weekly basis, there is a constantly moving target. Firms can stay ahead of the curve by training teams on the tools and creating groups of power users who are ready to implement LLM solutions in casework. It is essential to continually evaluate the potential risks and limitations associated with LLM use and work closely with clients to discuss suggested methods and applications. For those firms at the forefront of bringing AI-driven innovation to the sector, the imperative is to balance enthusiasm for integrating new and exciting technology against the required utmost accuracy and precision.
The views expressed herein are solely those of the authors and do not necessarily represent the views of Cornerstone Research.
