LLMs are more likely to give positive portrayals of large, global brands, according to a worldwide study. What does that mean for law firms hoping to secure visibility in ChatGPT output?
In late 2024, the Association for Computational Linguistics completed a comprehensive study to see how LLMs would discuss various brands of shoes, clothing, beverages and electronics, based on where the queries originated. Across product categories, LLMs, and geographies, the experiments found a “clear pattern” of brand bias; that is, “LLMs associate global brands with positive attributes and local brands with negative ones, consistently across multiple models.”
As the research conclusion stated, this “could affect consumer behavior and brand perception,” as it “could exacerbate challenges for local brands competing in a global market.”
Law wasn’t among the topics tested, but with Big Law “infiltrating” secondary markets and opening offices in locations like Salt Lake City and Nashville, an increasing number of law firms find themselves competing with global players.
Using my own market of Kansas City, I ran my own experiment to see if the research would hold true regarding law firm visibility in LLMs. Simply put: Do the robots think bigger is always better?
The Parameters
My search: “Give me a short list of commercial litigation firms in Kansas City.”
My reasoning:
- Searching for “law firms” instead of “lawyers” would surface legal brands instead of individuals.
- Commercial litigation is a field with significant talent in both small and large firms; in addition to dozens of strong local boutiques, our market has significant representation from the AmLaw 100 (eight firms) and AmLaw 200 (seven firms).
I performed the search on five platforms:
- ChatGPT: Paid
- ChatGPT: Free
- Microsoft Copilot
- Gemini
- Google AI Overview
The Results
To determine whether an LLM favored large or small firms, I evaluated the size of law firms it recommended in its first three results (who reached the podium, if you will).
There was a clear Big Law bias on three of the five platforms:
- ChatGPT: Paid. Here, the top three results went to an AmLaw 200 firm based in Kansas City, an AmLaw 100 firm based in Kansas City, and a local midsize firm. Average size of podium firms: 517.
- ChatGPT: Free. The top three results went to an AmLaw 100 firm outpost, then two AmLaw 100 headquarters locations. Average size of podium firms: 1,117.
- Gemini. The first three: an AmLaw 200 firm headquartered in Kansas City, an AmLaw 100 firm outpost, and an AmLaw 100 firm headquartered in Kansas City. Average size of podium firms: 950.
Meanwhile, there was a small firm preference on two of the five:
- Microsoft Copilot. All of the five results from Copilot were local small firms, ranging in size from 1 to 17 lawyers. Average podium size: 4.
- Google AI Overview. This was the longest list of results, with 17 suggestions. Small firms dominated these results, taking the first 13 results; the top three spots went to firms ranging from 1 to 14 lawyers. Average podium size: 7.
Of note:
- ChatGPT models cited sources, including firm Wikipedia pages and Chambers rankings, that naturally favor Big Law. It’s another reminder of the renewed relevance of rankings in the AI era.
- Microsoft Copilot’s exclusive use of small firms is particularly interesting given the tool’s popularity in large law firms, where it is bundled alongside business-as-usual tools like Word and Outlook. Copilot is in use at giant firms like Husch Blackwell, DLA Piper, and Clifford Chance; if their attorneys use the tool to find local counsel, they may receive reputable solos and small firms instead of their Big Law peers.
- Not surprisingly, Google AI Overview was the platform most subject to listing firms with aggressive SEO tactics, including one law firm seemingly based in Oklahoma but with significant keyword work for Overland Park and Kansas City.
What It Means
In this (limited, informal and certainly not peer-reviewed) experiment, there was a Big Law bias, with three of the five LLMs (60 percent) recommending large firms over smaller ones.
The discrepancy is not overwhelming, though; even the LLMs that favored larger law firms also included smaller firms on the list. And as legal marketers learn more about how these models are trained to scout lawyers and law firms, we can develop strategies to increase any firm’s visibility.
The Tale of the Tape: LLMs That Love Big Law
The Tale of the Tape: LLMs That Favor Small Firms
