Artificial intelligence (AI) and machine learning (ML) are increasingly influencing many aspects of the legal sector, including legal expert analyses. To determine how often AI and ML are used in expert analyses and identify what these tools are being used for, Cornerstone Research’s Data Science Center conducted a study in this area earlier this year. We examined the prevalence and applications of AI/ML in expert reports over the past five years, utilizing sources such as LexisNexis, Bloomberg Law, Law360, and public press.
Current Utilization of AI/ML in Expert Analyses
The research identified approximately 20 cases where AI/ML techniques were employed and documented in publicly available expert analyses. The primary applications included sentiment analysis, topic modeling, and clustering. Other instances involved computer vision, information extraction, and text comparison methods.
In several cases, the experts did not implement the AI/ML analysis. Rather, they relied on results provided by third-party data brokers or off-the-shelf AI/ML analysis tools. Importantly, none of the identified analyses incorporated generative GenAI or large language models (LLMs). When challenged in court, these AI/ML-based analyses were generally accepted and not excluded solely due to their use of Al or ML.
Future Outlook
Despite the current limited use, there is a growing trend toward integrating AI/ML in expert work. Advancements in LLMs have opened the door for enhanced capabilities in content and document analysis. For instance, LLMs have improved sentiment analysis by enabling more accurate entity and domain-specific evaluations.
It’s worth noting that content analyses don’t need to be limited to text; multi-modal LLMs and non-LLM approaches can classify images and identify target objects, which can be useful, for example, in identifying the appearance of certain features in marketing materials or specific products or logos in images posted to social media.
Document analysis at scale has seen significant improvements with LLM integration. In a recent case involving the review of over 75,000 SEC filings, we demonstrated that LLM-assisted workflows delivered true positive identification at a comparable level to human review, reduced false positives by 85%, and identified up to 67% more relevant examples. These findings suggest that LLMs can enhance the quality, efficiency, depth, and scalability of document analysis.
In topic modeling, LLM-based approaches offer more precise and interpretable results compared to traditional methods. Text classification tasks also benefit from LLMs' ability to achieve higher accuracy without requiring the curation of large, task-specific training datasets.
Looking ahead, AI/ML is poised to significantly impact expert support and testimony. Emerging areas include research and coding assistance, where recent advancements have enhanced capabilities. AI tools can assist in conducting and synthesizing research, such as academic literature reviews or industry analyses.
The increasing importance of AI/ML in expert work is also evident in the growing number of AI-powered tools and software being developed specifically for expert use. For example, various AI-powered research tools can assist in conducting literature reviews, extracting data from complex documents, and analyzing large datasets. Additionally, AI-powered coding tools can help experts write more efficient and accurate code, reducing the risk of errors and improving the overall quality of their work.
Potential Risks and Considerations
While AI/ML offers substantial benefits, experts must be aware of potential risks, including disqualification or challenges under the Daubert standard for misusing AI technology. The legal framework governing AI use in expert work is evolving, with no universally accepted rules currently in place.
A notable example is the November 2023 Celsius Network bankruptcy proceeding in the Southern District of New York. An objector's valuation expert submitted a 172-page report generated by AI within 72 hours, a task that might require over 1,000 hours for a human-authored report. The court excluded the AI-generated report, citing its unreliability and failure to meet admissibility standards. Key issues included a lack of sufficient facts or data, minimal citations supporting the opinions, and the expert's admission of not reviewing the underlying source materials.
In addition, LLM hallucination—incorrect or fabricated information presented as factually accurate—has been well documented and publicized. While hallucination risk may never be fully eliminated, experts can use methods to mitigate this risk such as prompting the model to generate a response from specifically retrieved context and manually verifying all facts and citations.
Best Practices for AI/ML in Expert Work
The benefits of AI/ML in expert work are numerous. AI/ML can enhance the quality, efficiency, and accuracy of expert analyses, freeing up experts to focus on high-level tasks and strategic decision making. AI/ML can also help experts to identify patterns and connections that may have gone unnoticed, leading to more comprehensive and accurate analyses.
To ensure responsible and effective use of AI/ML in expert work, several best practices must be followed. Experts must be trained and educated on the use of AI/ML tools and techniques and must understand the limitations and potential risks associated with their use. They must also be able to critically evaluate the output of AI/ML tools, ensuring that they are not relying on incomplete, inaccurate, or biased information. Lastly, experts must be prepared to explain and justify their use of AI/ML tools and defend their analyses in court.
Conclusion
While our research shows the integration of AI/ML in expert analyses is currently limited, its use is expanding. GenAI and LLMs are poised to create a shift in economic litigation, with experts and legal consultants increasingly leveraging these tools to enhance the quality, scope, and speed of their analyses. In the areas of research and coding assistance, the last year has seen some dramatic improvements in capabilities, including agentic AI tools. However, it is imperative to apply AI/ML responsibly, ensuring rigorous oversight, verification, and adherence to evolving legal standards.
Glossary
Artificial Intelligence (AI): aims to create systems that can mimic human cognitive functions like learning, problem-solving, and reasoning. It encompasses a wide range of techniques and algorithms.
Machine Learning (ML): is a sub-discipline of AI that focuses on data-driven decisions. In contrast to early AI approaches, where decision rules were explicitly coded, machine learning comprises a variety of algorithms that provide instructions for determining optimal decisions from provided data.
Large Language Models (LLM): a machine learning model whose understanding of human language was developed through extensive training on vast amounts of text data, enabling it to learn patterns and relationships within language. Many modern LLMs are generative in nature, allowing them to perform tasks like answering questions, translating languages, and writing content.
Generative AI: (GenAI): a subset of AI, GenAI focuses on creating new content (like text, images, or music) based on user input. This is different from traditional, non-generative ML, which typically focuses on predicting or categorizing data.
The views expressed herein are solely those of the authors and do not necessarily represent the views of Cornerstone Research.
