Editor’s Note: The eDiscovery field has reached an inflection point where professionals can no longer rely on static expertise to sustain their careers. This EDRM workshop, sponsored by HaystackID®, captured essential insights from industry leaders who understand that survival in legal technology requires more than keeping up; it demands getting ahead of the curve. The panelists’ emphasis on combining free resources with in-person networking reflected the reality that successful learning happens through multiple channels and genuine professional relationships. Their collective wisdom reveals that intellectual curiosity and continuous adaptation aren’t just nice-to-have qualities; they’re the fundamental skills that separate thriving professionals from those left behind. For anyone looking to thrive in their eDiscovery career, this discussion provided both the wake-up call and the action plan needed to stay competitive in a changing market.
In eDiscovery, outdated knowledge becomes a liability rather than an asset, making continuous education essential for career success and advancement. As organizations struggle with massive data volumes while trying to cut costs and minimize risks, legal tech professionals must anticipate technological changes, regulatory updates, and evolving best practices to maintain their relevance and effectiveness.
The discussion dove into how generative AI (GenAI) is transforming eDiscovery workflows, particularly in document review, while creating new competency requirements in prompt development and AI fluency. The panelists warned against viewing AI as an automatic solution, advocating Relativity’s David Horrigan’s “trust, but verify” philosophy and stressing the continued need for human supervision, quality control, and appropriate constraints.
Complacency in our field creates a dangerous illusion of security while the ground shifts beneath your feet. Professionals who rest on their current knowledge often find themselves blindsided when new technologies or methods suddenly become industry expectations rather than optional skills. The gap between staying current and falling behind can widen surprisingly quickly, leaving those who assumed their expertise would remain valuable scrambling to catch up in a market that has already moved on to new standards and approaches.
Hi everyone. Welcome to today’s workshop, “Building eDiscovery Expertise Where Education Begins and Never Ends.” I’m Mary Bennett, and on behalf of HaystackID and the EDRM, I am excited to have you here. And before we begin today’s program, I would like to take a moment to pause and acknowledge the profound loss that many of us in our community have suffered this week with the sudden passing of our dear friend, Kaylee Walstad. She’s the Chief Strategy Officer of the EDRM, a friend to many. I met Kaylee early in my eDiscovery career, and her energy, as so many of us know, is just magnetic. She’s been a mentor to me, a good friend, through both work and personal accomplishments, always in my corner, and even during the tough times. As we all know, she was always, always, always there. She was such a bright light and will be very, very missed. We will keep her in our thoughts and just want to share this today, in honor of her, knowing that we’re thinking of her and her family. Turning it over to anybody else on the panel who has any words about Kaylee they’d like to share.
Mary, I’ll jump in. I think that remembering Kaylee today is probably incredibly appropriate, given our topic, which is education and eDiscovery. There are very few people who have done more for education in eDiscovery than Kaylee Walstad. I think she always thought of herself as a connector, and she certainly was. She brought many people together. She helped bring this program together, and over the years, she has been actively reaching out to everyone. She had a long career on the quote vendor side of eDiscovery, and she brought that experience to networking for the entire community. She brought so many people together. And when you talk about Kaylee, there are so many people who said, “Wow, I had confidence in myself and my abilities because Kaylee helped me have that confidence in my abilities.” And there are so many people whose lives she has touched. There are also a whole lot of people who know about eDiscovery because of her amazing work here at EDRM, as well as all the things she has done over the many years. She will be missed. And clearly, this program, along with many others, is a tribute to her.
Thank you, David. Yes, she definitely made you feel so powerful. I would talk to her for five minutes, and I’d think, ‘Wow, I can do anything.’ It was just so amazing.
Yeah, that’s what I recall, just her seemingly unlimited enthusiasm and spirit and energy. She was always on the go and was always available as a resource for people, not just on an institutional level, but also on a personal level. If you had a question or a concern, she always made time to offer you some good advice and a pat on the back. As a Boston Irishman who was prone to our national characteristics of melancholia bordering on moroseness, her just complete and utter enthusiasm was always an ongoing mystery to me. There were a couple of great things written by Craig Ball on his blog yesterday, and Doug Austin discussed them this morning. And many people use the phrase ‘speechless,’ which in our business is really saying something when folks reach out and say, ‘I just don’t have any words.’ The only other comment I would make is that there has been an amazing response to the postings that have been out there from everybody in the business. And I’ve said this before, when somebody we know passes, don’t wait until someone you love passes away to tell them that you love them. It’s great that we have all these words for Kaylee now. I don’t downplay that in the least, but reach out to folks and let them know that they’re important to you, that you care about them, and do that today. Don’t wait to do that. We must maintain that interface, that interplay, if you will, between us all.
I don’t think I could say anything else better than what Tom and David have just said. So all I’ll say is that she was just a larger-than-life personality, and I can see her in my mind’s eye at any number of conferences standing across the room, holding court, huge smile on her face, just somebody that’ll be just so, so missed by all of us, I think.
Thank you. Thank you all. Holley, I’m going to turn it over to you, my friend, to just share a little bit about the platform and resources.
Mary, it’s great to be back with you as always.
Mary Mack will not be joining us today. We are thinking of her and the Walstads in our thoughts. And Tom, thrilled to have you here from Boston. As you all noted, perhaps we’ll see that accent emerge. We will see. He is the Director of the Gulf Coast Legal Technology Center. He’s a nationally known consultant, speaker, and writer in areas such as computerized litigation support systems. He’s from New England, which is exciting. Graduated from Johns Hopkins in 1972 with a BA in political science. So Tom, excited to have you as well.
Well, thanks. Enjoy being here.
And I am Mary Bennett. I wear two hats here today. I’m the Senior Director of Content Marketing at HaystackID. I am the Senior Director of Content and Community Initiatives at the EDRM. I am very passionate about storytelling, education, and legal tech education. I have been part of the industry for several years and continue to learn and grow. I am extremely excited personally about this program and to hear from the wonderful panelists here today. So, with that, let’s get to the conversation, which is the bread and butter of today. Tom, I’m going to kick it off with you, my friend. How do you define education and eDiscovery? We talk about it a lot. Is it rooted in formal training, case law, tech proficiency, or all of the above? None of the above? What does it mean to you?
Probably all of the above. Coming from New England, as I do, the birthplace of public education in the United States, we’ve always had a real focus on educational systems in New England. That’s based on the philosophical concept of pragmatism. A school of thought initiated by William James and John Dewey, a renowned educator, posits that practical, results-oriented approaches are the most effective way to educate. Education isn’t limited to classrooms, books, and lectern-based teaching. It’s about in our world, especially getting out there, getting your hands dirty, working with the data. And I think that’s especially true in our field, where things change so rapidly. I mean, amazingly rapidly, that practical experience and a practically oriented approach to education is the best way to proceed.
I think that’s a great point, too, that education has always been important in any industry you’re in. However, with so much change happening, it’s essential to make it part of your daily routine to absorb what’s going on and apply it in practice. This involves putting pen to paper to effectively utilize those skills in a rapidly changing environment. So thank you, Tom.
Mary, underlying that too, if I can just jump in on that thought. I think everything that you and Tom have said is true, but I think driving that educational zeal is this concept of intellectual curiosity. Find something that interests or excites you, and pursue it. That could be data privacy challenges. That could be the use of a given tool. That could be how messaging data works, what the metadata looks like, how we collect it, and how we process it. So whatever it is that drives your area of interest, going right after that and really digging in and maybe even becoming the go-to person for that specific thing, that’s driving that education in eDiscovery. And that’s true of so many things out there as well. But I think it’s really diving into what, like I said, interests you that drives that education. I talk about all of eDiscovery is really that. I mention that, and I’ll date myself a little bit, but I graduated from law school in 2006, which is the same year the federal rules of civil procedure were amended to include electronically stored information. And so at that time, even getting into eDiscovery at all, because there was really no roadmap then, I was having intellectual curiosity about this new developing problem and where’s that going to go, and what does the overlap of law and technology look like? So getting into that at all requires that level of intellectual curiosity. And then I think within that, what interests or excites you is where you want to drive that effort towards education and learning.
And making space for that. Because we’re all so busy, it can be hard to sit back and be like, What does light my fire?
Well, Mary, I can’t agree more with everything that Tom and Ryan said. And the answer to your question, I think of course, is all of the above. All of these resources were important. And as far as a philosophy goes, I’ve said more than once, to entertain is to educate. The idea behind that is not to be flippant and superfluous, but rather to be substantive. When we discuss these topics, especially during the eDiscovery State of the Union session at Relativity Fest, we refer to it as substantive fund because it must qualify for CLE credit. And so, there’s substantive law there, but I think people remember more if they can connect with it and if they get a good laugh out of it or find it memorable. When I’ve written articles over the years, especially in eDiscovery, what is germane to holding a lot of people think is just what the eDiscovery holding is. However, if you read a case law summary or the actual case in eDiscovery, the decision may have been an order or a holding that applies throughout the case’s life, and there could be multiple decisions in a single case. Often, you won’t be able to determine the fact pattern. I think it’s essential to read the entire case and go beyond what you see in the case filing to learn about the parties involved, the nature of the dispute, and the context surrounding it, as people will remember these details. And when you’re going down M1, 5100 is a very well-known self-collection case by Chief US Magistrate Judge William Matthewman. When you think about it, I always remember it as the Jumbo supermarket case because you go in and find out it’s about a small chain of grocery stores in South Florida, the Jumbo supermarkets, and then you learn about some of the allegations related to age discrimination. And so you always remember, oh yes, of course, the self-collection concepts that were in Jumbo supermarkets. And so I think those sorts of things, remembering what the fact patterns were, remembering what the story was, and I think it just helps you remember, and it’s all about education, and I think once again, that educates you.
Substantive fun. That’s a good way to brand it, David. I like that. What about the specific resources? Let’s get into what your go-tos are. What do you keep up with regularly, based on your interests, based on your roles? What do you use as your north star to stay in the know? Ryan, why don’t we start with you this time?
Sure. So, the resources that are out there, to put it bluntly, are oftentimes fantastic and often free. So, stuff that you can get without having to pay for, whether it’s webinars like this one today, whether it’s other content that’s out there, white papers, and that sort of thing. Anything that you can get your hands on that meets your area of interest is where I typically tend to focus my own content search. I also really like, and I’m going to go outside of the digital tech area here, but I also appreciate in-person training sessions. Now, you will often have to pay for this, but generally speaking, with most roles, there’s a budget or allocation for education, training, and so on. If you can get just a little bit of that and go in-person to a training session, meet the other people that are in your field, meet the other people that have that same area of interest, ask questions, see what their specific challenges are, see what comes up in that context, make friends that in-person training session when you’re paying for these things can be invaluable and many, many times more valuable than whatever the cost is to attend something like that. So, especially post-pandemic, and given the current state of the world, with everything moving and changing at a rapid pace, in-person human connection is truly invaluable. And so, really looking for opportunities like that, I think, is incredibly important, and that’s what I seek out myself for sure.
Definitely. I know, David, back when I was at Relativity, I was studying for the RCU, and just sitting at my desk was one thing. As an English major, it took me a little while to put one and two together, but I remember sitting with other people studying for it and getting into the platform to actually use it. It just clicks in a different way. I think that in-person interactions can be invaluable in many different ways.
David Horrigan
Yeah, there’s no question about that. And I don’t want to sound like a shameless plug, but as the Legal Education Director at Relativity, we’ve got a lot of resources at Relativity, be it what Mary did with the certified user exam, what we do in legal education at Relativity Fest, the webinars that we do, and the data discovery year-end review. And I might as well continue on this theme of shameless plugs, because our hosts at EDRM offer not only this webinar, but also many other webinars, including the Generative AI Court Rulings Database, numerous databases, and various working groups. I mentioned Judge Matthewman before; there’s a great EDRM webinar coming up on generative AI and hallucination sanctions. It will feature Judge Ralph Artigliere, as well as Judge Matthewman and Professor Bill Hamilton from the University of Florida. That one is a don’t-miss webinar that EDRM is hosting. To Ryan’s point, a lot of these are free. Another great resource. I’m going to continue with the shameless plug because this is someone who is a colleague of Ryan’s, a colleague of Mary’s, and a close personal friend of Holley’s, I might add, and that’s Rob Robinson, who’s not only with HaystackID but also with ComplexDiscovery. ComplexDiscovery has a wealth of information on it. Rob Robinson’s experience in the industry spans many years, and he leverages that expertise to cover both domestic and international developments. And you should check that out. I could mention Doug Austin, as well as many other resources, but I need to reserve some of these for Tom to discuss.
Tom O’Connor
Oh, I was just going to say ditto. I was just going to say ditto. Well, I definitely echo some of the same comments. Doug Austin’s right at the top of the list for me, followed by eDiscovery Today, and then Rob Robinson with ComplexDiscovery, which I’ve been following for longer than I care to admit. The EDRM, without a doubt. Bob Ambrogi’s LawSites, I believe, is an excellent source of information, and I enjoy the columns in Above the Law, which feature a series of different columnists who write about a variety of subjects, often providing insightful commentary. However, I would like to echo the comment about in-person interactions. We just came back from ILTACON, which is a great, great in-person educational resource. I think user groups in general are great. Relativity Fest is an awesome resource. This and EDRM with their webinars. I just learned something. For example, I learned how to correctly pronounce Judge Artigliere’s name, even though I’ve known him for a number of years. Very embarrassing. So thank you for that, David. There is a wealth of free resources available. All of the ones we’ve mentioned, I think, are well worth signing up for. Most of them offer a weekly feed, which is a convenient option. Alternatively, if you prefer to receive on-the-fly RSS feeds directly in your mailbox, you can do so. I would also mention the last one, which is strictly case law. It’s not free, but it’s reasonably priced, and that’s Kelly Twigger’s Minerva26 for her assessment of case law in our space. Doug Austin, I know, uses that. I’m sure several other people use it as well. Really, really good resource.
David Horrigan
Tom, that’s a great one, and I use it too. Minerva26 is a great resource, and for those of you who are just getting into eDiscovery, Minerva26 offers a wealth of information beyond case law. The Academy is a portion of it that has Kelly’s case of the week. It’s got articles. In fact, there’s an article by Judge J. Francis, whom many of us know is associated with it. There are many useful articles in there. So, you have the case law in one section, the academy in another, and a whole bunch of resources, including many of the rules. And so that’s a great one to have for an in-depth dive. That’s eDiscovery specific. Cornell’s Legal Information Institute is great for actually understanding the rules, and that’s another free resource. You go right on there. And actually, it’s great that if you Google, say, Federal Rules of Civil Procedure 26, where the proportionality six-prong test is, Cornell is usually one of the top hits, and that will come up. That’s a wonderful free resource for many things, and we’ll definitely help you out.
Mary Bennett
Great. I’m going to jump a little bit ahead here, panel, because I think this lends itself nicely, Ryan, to thinking through people’s different stages of their career. I’m sure many of us on here are at those different stages. Perhaps you’re new to the industry, or you’ve been in it for several years, trying to uplevel, or maybe you’re just looking to stay sharp. How do you think they should approach education at those different stages, and do you think different resources are valuable at different times?
Ryan Costello
I do. Thank you for the question, Mary. We’re talking about junior-level careers, mid-career level, and then more senior levels. Starting with those in the early stages of their career, I would recommend again looking at the area that really interests you and becoming the go-to person for that. So let’s say it’s data privacy, sign up for the International Association of Privacy Professionals. Learn everything you can about that subject area. Again, if that’s what interests you, then look at where that overlaps with eDiscovery and be the go-to person. When a client has a cross-border discovery matter, they need help and guidance, or at least some insights that they are looking to someone for; you become that person, the go-to expert in that subject area. Or, as cited earlier, consider the example. If it’s messaging data, collecting messaging data, processing messaging data, that’s what interests you. If that’s what you’re into, being the go-to person for that. So, really driving on that one area of expertise for which clients, colleagues, and others will rely on you specifically. I think that’s a great way to kick off your career at that stage. As you get to more of the mid-career level, I think you broaden the scope at that point. So you begin doing a little bit more of the cross-industry focus, make sure you’re at these in-person training and conference opportunities, being there, meeting people, getting to know a wider swath of the industry allows you to grow within your career and move outside of that niche that you’ve created for yourself to broaden your horizons in that respect and again, continue to grow as professional. For those in more senior positions, presumably, you’re working with more people, having worked with many more, and you’ve seen things evolve over time. I believe that learning the skills of people management and working with diverse individuals, particularly those from different generations or age groups, and understanding their approaches to problem-solving and work, makes soft-skill training extremely important. Additionally, you can become a valuable resource in your more senior position, leveraging the knowledge you’ve gained over time and how you’ve witnessed changes and evolutions. You may also offer predictions or insights on how to navigate industry changes and adapt to them, among other things. At that point, you become the training resource for others who are coming up in their careers, focusing on people and how they work and operate, and so on. So, I don’t know if that quite answers your question, but that’s what I would advise or guide you in terms of where your focus should be at each stage of one’s career.
Mary Bennett
So starting out, it’s being a sponge in your area of interest. Learn everything you can about whatever that might be. As you become a little bit more junior, build your network, meet the people who are going to help you accelerate, and then I would say mentor others and really fine-tune those people skills to build the next layer. That’s really great.
Tom O’Connor
I think there’s another distinction to be made, though, and I agree totally with everything Ryan said, but part of it also focuses on what your role is. Are you an attorney? Are you a paralegal? Are you an IT professional? Where do you fit in your organization? If you’re an IT professional, you’ll likely focus on obtaining several technical certifications that may be available. If you’re a Microsoft shop, you want to obtain some of the Microsoft certifications available and stay up-to-date with what’s developing there. And they have a clearly defined path that you can progress through. If you’re a paralegal working with Relativity, there’s a host of Relativity certifications you can go through to be part of your learning curve, working up to the always elusive Relativity Master. Right, David?
David Horrigan
You got it.
Tom O’Connor
It’s much sought after but hard to obtain. But look at what’s available within the specific scope of what it is you’re doing, because there may be a variety of specific certifications and trainings that are available within your job description.
David Horrigan
Hey Mary, if you don’t mind my deviating a little bit. In the Q&A, I think Mohammed Rizvi’s got an excellent question. Given the advancement in AI, do you think review expertise will be relevant in the coming years? I think absolutely. And the real lesson is in some of the case law that has emerged; it hasn’t been reviewed in the eDiscovery case law yet. I’m sure it will very soon. We haven’t seen it yet. However, one interesting hallucination case is Mavy v. Commissioner. That’s a social security case out of the District of Arizona. And it really goes to show that this is a partnership between humans and technology, and humans will always have to play a role. And when you see a lot of these cases, as we discussed, Tom mentioned at ILTACON, and I recently wrote an article in Legaltech News, I referred to it as the Lamborghini Doctrine of Hallucinations in AI. And basically, the idea is that generative AI is like a Lamborghini. It is an amazing tool with numerous great features. You can potentially also drive it off a cliff if you don’t have seatbelts and read the owner’s manual. And in a lot of these cases, such as Mavy and the first one, Mata v. Avianca, there was nothing wrong with the tool. And the courts continually say that there is nothing wrong with using generative AI in legal matters, but you have to use it as a tool, not as the Staples easy button. And there were 19 cases cited in the Mavy decision, many of which were actual citations. Generative AI generated the actual citation. Some of them were not. There were a few that were just completely fake citations. So you’re always going to need that human in the driver’s seat, and certainly it’s going to have an effect on a review. Many would say that it would make it better. And to Mohammed’s question, I think one thing that generative AI does that early forms of technology-assisted review didn’t do was to give you what they call the citations. This is not a legal citation; it’s the citation explaining how the GenAI came to that decision. So as long as you treat it as a tool and not the Staples easy button, I think it’s just an exciting time. And to Mohammed’s point, it will have an effect, but humans will not be excluded from this process.
Tom O’Connor
No, I couldn’t agree more. David. There was a great article, I think it was by Tara Emery, that you may have seen recently, where they discussed the interplay between human review and the use of AI, and how humans direct and interact with it, creating a hybrid model. I think AI is going to speed up and enhance reviews, but replace human interaction. And of course, as the federal rules require supervision by an attorney who’s filing a particular motion, that’s not going to go away. Not at all.
Ryan Costello
Can I add a point to this, too, Mary, just to get one more thought out on this? I mean, the review expertise will remain relevant, but it will change for all the reasons that Tom and David have just explained. The human element of generative AI solutions and tools is critical, but the review part of it will change. So instead of reviewing document after document, you are crafting a prompt and working on crafting prompts and doing it so well that you’re reducing the amount of review because what the generative AI is able to give you is so on point to the body of documents that meet your relevance criteria. That aspect of being able to craft prompts like that will be a skill and an area of expertise that will drive review. Your review expertise and experience are still extremely important. It’s just changed in how you apply it and what it looks like.
Mary Bennett
Because that experience will help you craft really effective prompts.
Ryan Costello
Absolutely.
Mary Bennett
That’s key. I’m going to take this next question to you, too, Ryan, given it’s about information governance, which I know is in your realm. Mohammed also wanted to know if it is fair to assume that IG will be the most important area in the coming years.
Ryan Costello
I’m slightly biased about what I’m going to say on that topic because it’s my current area of expertise and focus right now. However, there was a Gartner study from a few years ago; although it may be slightly out of date, I believe many of its findings remain relevant. They stated that the volume of digital business data stored across all industries, sectors, and verticals is growing at a rate of 40% to 60% per year. We’re talking about huge volumes with exponential growth over time. Looking at information governance, archiving, and various solutions in this area can bring down storage costs, lower risk, and ultimately reduce the eDiscovery spend associated with it. So, just managing those volumes of data effectively. Additionally, there are various tool solutions and approaches that you can apply to that data classification, which is one of the key points. But making sure that, and this goes to a lot of other things that I think we’ll talk about in a minute. It’s crucial to manage overall costs and the data itself effectively. I do think the IG area will grow in increasing importance, also because we’re now looking at evolution and tools that, in the very near future, may enable us to collect and even process data in place. Therefore, the EDRM will change slightly and adapt to improvements in technology and other factors. And again, all of that is looking further left and right now in the EDRM in terms of the importance and value that one can bring to that. So definitely going to be more and more important in the coming years for sure.
Mary Bennett
Why don’t we stay on that train there, Ryan, with the left side of the EDRM? Why is that increasingly important, and what foundations do people need to really engage at that earlier stage of discovery?
Ryan Costello
Sure. Therefore, this has some bearing on the work we are doing with the legal data intelligence group. So, if I can use that as a springboard to discuss it a little bit. So, LDI, or Legal Data Intelligence, is a group of professionals working in this space or outside of it across different industries and service providers, and sectors to bring vocabulary and a framework, as well as a series of best practices, to manage all of that data. What we’re looking to do is help organizations reduce what we call ROT, so redundant, obsolete, and trivial data, by bringing out the SUN, which is the data that’s sensitive, useful, and necessary. Considering what can be done to manage the volumes of data I mentioned a moment ago, doing so effectively really reduces risk and overall cost to the organization, as I mentioned earlier, whether that’s through storage or Ultimate eDiscovery, which allows for further downstream cost savings in the process. And so we are again, as I mentioned, LDI series of professionals who’ve varied experience in the space, developing a set of guidance documents, best practices, use cases, specific examples about what that looks like across various industries, and how one can manage all of that data effectively, really bringing value to the overall legal process. It’s hugely important and very relevant to a lot of what we’re examining across this space right now.
Mary Bennett
Getting rid of the ROTs and embracing the SUN. Those are good, good acronyms we have here. David, do you have any thoughts on LDI or other informal or formal groups?
David Horrigan
I think Ryan gave a very articulate description of what LDI does. The only thing I would add that’s important is that when you look at LDI, there are 17 models and use cases. And what LDI can do is allow you, as an eDiscovery professional, to take your tools and knowledge and have them in a much broader context, for lack of a better term, with the fish seeing the sun. And it’ll really help you expand your career. Scott Milner of Morgan Lewis speaks very eloquently on this. He’s involved with LDI. And it is the idea that, whether it is data privacy, as Ryan mentioned earlier, the IAPP is a great resource for many of the things you can do with data privacy, such as contract review. These skills that you have developed through your study, your learning, and your career in eDiscovery can take you to a whole host of areas. And LDI is a great resource for helping you advance your career in that way.
Mary Bennett
Thank you, David. Tom, I want to go to you. Throughout your career, how has your approach to education shifted? Where do you think people should focus on upskilling right now? How has that changed in the last several years, in terms of where the focus needs to be, or has it changed?
Tom O’Connor
Well, it’s definitely changed. Mary Mack and I were discussing this in a conversation a week or so ago. She and I worked together years ago, in the prehistoric days of eDiscovery at Fios in Portland, and education was then basically the old person at the lectern. You had to go to the firm, give a presentation, and possibly speak at a bar association gathering. The first thing that has changed, of course, is the internet, which enables the distribution of information widely over the internet, allowing for webinars like this and bringing the classroom to the desktop. That’s made an immense difference in the availability of all sorts of educational tools. And I think the second thing that has changed is, and I’m not sure this is whether this is for the good or not, but the taking of that education, I’m not even sure we took it, from educational institutions, and I’m speaking specifically of law schools who really haven’t addressed for the most part, education in this field. However, it has opened up a broader expanse of organizations, such as the EDRM, and companies like Relativity and HaystackID, which can provide this education to people. It provides people not only immediacy but also a wide range of choices in an open market, if you will. So, yeah, I think that’s not just our space education in general; you can now earn a college degree or an advanced degree online. The internet has changed everything.
Mary Bennett
Truly. And I do want to address Amanda and Peter’s questions on AI, because I think we’ll never escape the impact of AI on it all. But David, I’d like you to talk a little bit about law schools here, since you are a guest lecturer and work with numerous law schools and bar associations. Relativity has an academic partner program that works directly with these schools. How do you envision these schools contributing to eDiscovery education and the upskilling of talent?
David Horrigan
Sure. Thanks, Mary. And Tom and I had a friendly debate in one of our prep sessions for this broadcast about the role of law schools. I think law schools are making a strong effort in this regard. And not to discount Tom’s point. Tom’s point was that other organizations need to pick up the slack because law schools haven’t really done a great job. Tom, am I misquoting you on that? But that’s the general gist of it.
Tom O’Connor
No.
David Horrigan
Okay. Just want to make sure I’m not putting words in your mouth. And Mary, as you mentioned, I give law school guest lectures. In fact, this fall, with the Academic Partner Program at Relativity, I’m serving as an adjunct professor at Duquesne University’s Kline School of Law. We will bring the students some of the Relativity training, and I will also cover the case law and rules. So you’re combining the hands-on training that you get by actually using the tools with the academic training of learning the case law, getting to know the rules. And our good friend Professor Bill Hamilton at the University of Florida, shameless plug number four, I think, on this program, because I went to law school there, and Tom and I have been involved in that conference for a long time. Mary and Kaylee have also attended it over the years. Ryan and the HaystackID crew have been there before. Bill Hamilton is one of the real messiahs, for lack of a better term, in legal education and the law school sector. And of course, Professor Hamilton was a practicing attorney. He was a partner at a major law firm for many years, bringing that experience to the program. There is a major conference held every spring, where professionals from all over the industry gather. That is not to discount the great work that the HaystackIDs of the world, such as organizations like EDRM, do with all the free resources. As for the academic partner program, the latest count was around 115 or 116 schools. Some of those are paralegal programs. Additionally, it is worth noting that you don’t need to attend law school to become an eDiscovery professional. Some of the paralegals in the program possess incredible technical skills that significantly enhance the process. In fact, we conduct a general counsel survey every year at Relativity in collaboration with our partners at FTI. And every year, it emerges that paralegal skills are perceived to be better than those of attorneys in some technical areas. So, whether it’s a paralegal program, a technical program, EDRM, or what HaystackID offers, admittedly, what we offer at Relativity, there are many resources available. And, to Ryan’s excellent point, many of them are free.
Mary Bennett
And I think it’s interesting that you always hear on the EDRM Illumination Zone with Kaylee, Mary, and Holly, that so many of us fell into this industry. That’s just the way it was. However, it’s now even more interesting with law school, given the awareness that this is not only a viable field but also a great one to be in. People will proactively be raising their hands to come in, which should be interesting. So thank you, David and Tom. I want to revisit Amanda’s question regarding AI, as I think many folks here likely work with attorneys. So I think this might be relevant for many folks. It says her office recently brought in AI software, and the degree of hallucinations has left a bad taste for some attorneys who have tried and want to use it. Are there any helpful tips to help keep those users still eager to use and embrace the product?
David Horrigan
Mary, I have a rule regarding CLEs that we don’t discuss commercial products unless they’re specifically relevant. And I’m concerned that I sound like an infomercial here, but to Amanda’s point, you have to buy software that is expressly designed for use in the legal field. If you examine all these hallucination decisions, and we now have over 300 of them, I am not aware of a single one that was made using a legally based, closed-model AI software designed for legal purposes. In Mata v. Avianca, it was ChatGPT. And that’s not to disparage ChatGPT, but it is not designed for a legal process. Amanda has just piped in using the product she has. I’m certainly not going to disparage that product, but it goes back to the point that Ryan, Tom, and I made in response to Mohammed’s question: you can’t take out the human element. It is not going to be perfect, but most of the products have something designed to control for the hallucinations. You need to verify the citation. We had the annual Discovery Litigation Support Roundtable on our program at ILTACON. We had some fun by including a photo of the bait stamper and conducting a quiz for the audience to identify it. I included a pocket part from my own library. And I went to law school even a little bit before Ryan did, and Tom was in law school also, so I’m sure the three of us can remember the pocket parts. You had to go by there. And I don’t want to steal Tom’s fun story, but he was talking to someone at ILTACON, and he mentioned you got to shepardize these cases. And the response was, What’s shepardizing? The thing about it is, you can get led down the wrong road with something that we had way before GenAI. You need to verify the citations, and you can’t rely on these tools as the Staples easy button, as I mentioned earlier in our program. Sometimes, if you were going through a shepard going through a pocket part, it could lead you in the wrong direction. Now obviously, the old books, dusty or not, are not going to be hallucinating, but they could bring you to an incorrect conclusion, which is why you always had to check the citations. So, Amanda, to your question, it can speed up your process and point you in the right direction, but you’ve got to stop that Lamborghini and take a look at the citations. And I think that will really help in terms of utilizing many of these tools in a closed universe.
Tom O’Connor
Hey, two quick points, and I totally agree with David’s observation if we’re talking legal-specific AI versus the wide, wide world of AI. But there are two things. You’ve got to remember that if you ask a computer to do something, it’s going to try to do it. And so, if you ask an AI program to tell you how many Lamborghinis drove off the cliff in Sausalito last year, it will depend on two things. Number one, first and foremost, is where is it looking? What is the data that it is looking at? Is it examining just your data within your firm, behind your firewall, possibly even on your specific case or set of cases, or is it examining the entire internet? You have to put that guardrail in place. You have to ensure that the guardrail is in place. You know where it’s looking. The second thing is to specifically state in the program, ‘Don’t make something up.’ If you can’t find a specific answer, that’s okay; please let me know. Okay? You need to put that limit, those “guardrails” as they’re called in the business, on how the program is going to operate. In the legal world, many of those guardrails are built in, but it’s crucial to ensure they’re in place. The computer will attempt to answer your query, and when it can’t find an answer, it will generate one unless you tell it to stop. If you can’t find something in this specific set of documents, don’t go scouring the World Wide Web looking for it. Stop and just tell me. That’s where I’ve gotten, and we’ll make another query.
Mary Bennett
Yep. I think that’s a great point. Peter also made a great point about the varying views of AI and people’s perceptions of it, based on their position within the organization. In-house teams view it as a way to save costs, while outside counsel might be more hesitant to use AI, and as a result, the staff may not have sufficient training to use it correctly. So, in your opinion, when do you think those at the top will really commit to investing in AI and ensuring there’s proper training so that people are using it effectively and everybody’s more on the same page?
Ryan Costello
Therefore, the reduction of cost will always be a driver in the adoption of any tool or solution. There has been a push across many companies and organizations to allocate a certain amount of budget to technology improvements in a given year. And so GenAI solutions nicely check that box. I think that simply deploying technology, though, addresses only some of the great points that both David and Tom have made. Is it fit for purpose? Is it the right tool for the right job? Do we have some level of human verification or QA, quality assurance, or analysis and oversight to ensure what it’s doing for you is effective? Are you checking the work? We need some level of verification in place in order to do that. And I think it’s essential to hammer home that point to those who want to adopt a solution, emphasizing that this will reduce costs and save time. However, here are the guardrails that need to be in place for this to work. We’ve also vetted numerous different tools, and we think this is the best one for our specific purpose, as well as for that other reason. And so, just doing the necessary homework behind this and having a verification or QA process in place by humans is a huge part of initiating and working with these tools effectively.
Tom O’Connor
A couple of things I think will really drive pricing down. I totally agree with that point about cost. The price of a commodity will impact how much it’s used and accepted, especially in our business. However, remember that GenAI currently utilizes LLMs, or large language models, which use tokens for pricing. It’s not the old per-gigabyte model of pricing, and people often don’t understand that, including how tokens are structured and how they impact the pricing. Microsoft introduced a smaller language model designed for smaller amounts of data at a lower price point about a year ago, and they’ve been quietly building on it and revamping a lot of what they’re doing within the operating system. West announced at ILTACON last week that they’re also considering a small language model. I think the introduction of small language models will start to drive down pricing for some users. For some users. Lexis introduced something, they didn’t introduce it, I got to say several private vendors have been doing this for a while, but Lexis said they’re going to have an option where if you want to use AI, it’s going to give you a choice of which AI engine to use, so you can say, I want to pick out of these three. I think the market is moving towards driving pricing down in ways that make it more affordable and thus more attractive to us.
Mary Bennett
Great. And Mohammed, I hope that also answered your question on whether this is financially sustainable.
Ryan Costello
Can I add another point to that?
Mary Bennett
You may, Ryan. Go ahead.
Ryan Costello
Because there are some great questions here. I just want to have a back-and-forth discussion with the audience. However, to add to that point, Mohammed was asking, ‘Is the current state of GenAI solutions financially viable?’ Are they sustainable when compared to actual human doc review? And I think that they are, but I wanted to make a broader point: the human element of document review, specifically the actual review of documents and bringing in contractors and others to do that, is the most costly element of the eDiscovery process. As soon as a solution emerged that could reduce those costs, TAR, several years ago, when examining GenAI solutions, found that people were immediately interested in what they could do to lower those costs. And again, for all the points we’ve made before, these solutions will ultimately be viable and are going to change the way that review is conducted, but it needs to move at a pace where it could keep up to have that verification, those guardrails ensure that people are reading the manual and the Lamborghini and everything else. I think financially, it will be viable, but again, we’re not there yet. Those at the top who are making the decisions to use these tools need to be aware of this and educated on the limitations and requirements to use them effectively.
Tom O’Connor
Remember, this is an incredibly new field. I mean, how long has OpenAI been around? A year and a half, two years?
Mary Bennett
Yeah, I think coming up on two.
Tom O’Connor
Yeah. We’re seeing a massive compression of the time from introduction to market to the development of it becoming better. I’m reminded of the early days of OCR. Remember when OCR first came out, and it was like, yeah, well, it’s 90% effective, and it’s $5 a page, and nobody was going to use it, but within a year and a half, it became cost-effective, and now nobody even thinks twice about it, it’s just, it’s part of the framework.
Mary Bennett
David, why Lamborghini, not like a Maserati or Porsche?
David Horrigan
Harley Davidson, because that’s what came to mind, and I was thinking of many, many different topics, and to be in complete candor, my son had been talking about a Lamborghini, and so that’s what came to mind.
Mary Bennett
Lambo is a good car. They’re all great.
David Horrigan
And it’s funny, the original title of that article I had was, Don’t Blame the Paralegal or the AI. What I find horrible is when someone is caught for something and tries to blame someone else. And in one of these cases, the attorney had mentioned to the court, oh yeah, by the way, it was the paralegal who drafted that opinion and they used the generative AI when in truth they all had, but it wasn’t clear whether in the case the attorney was actually trying to blame the paralegal or having a moment of candor and saying, look, here’s what’s going on, because I didn’t know. Then I decided, let’s change that. It’s pejorative anyway, and so that’s how the Lamborghini came to mind.
Mary Bennett
All right. I was curious. Thank you. Tom, not a shameless plug, but a shameless plug for you, my friend. You do case law webinars that are such an essential resource for so many, whether someone’s an attorney or not. Why do you think it’s so crucial to keep up? What are the judicial decisions if you want to be effective in your job in this field?
Tom O’Connor
Oh, because so many are coming out so often, I mean, I participate, and I don’t do it. There’s a monthly webinar hosted by the EDRM, and Doug Austin is the actual on-site host. The panelists are Mary Mack, Judge Andrew Peck, who is well-known in this field and is now retired, but still with DLA Piper, and me. And so we do that once a month, and we typically handle six to seven cases. There’s no problem finding six new cases every month that cover striking, in-depth issues in this field. If we expand it to two hours, we could easily handle 15 cases. The problem is, and that applies to Minerva and Kelly Twigger’s offering as well. New cases are being reported constantly. Unlike anything I’ve ever seen in my experience, David, I’m sure we never used to see 20 new constitutional law decisions a month. It just didn’t happen. These things are everywhere, so you need to stay on top of them. In addition to the webinars we conduct, there are also a number of case-specific or regular monthly, quarterly, and other presentations by other individuals. You can keep an eye out for webinars on a regular basis. I think EDRM did one just yesterday. It wasn’t case law in general, but rather case law related to hyperlinks.
David Horrigan
Kelly Twigger was on that one along with Mary, and that was like, yeah, whoever thought you’d have that narrow a topic. And so to your point, Tom, a veritable cornucopia, as I say, far too often.
Tom O’Connor
A plethora. It’s a plethora.
David Horrigan
There you go.
Mary Bennett
I used the word ‘plethora’ in a team meeting the other day and got corrected. Took me a moment. You do learn something new all the time.
Tom O’Connor:
Well, it’s a movie reference too, as David knows it.
Mary Bennett
Oh, okay. I’ve been saying that word incorrectly for 32 years. So, in the interest of time, we have two minutes left, and we have so many great insights. Perhaps we should just close here, Ryan. We’ll kick it off with you. Where do you see your one area for the biggest demand for upskilling? If you could give someone one area where you’re like, ‘That could be it,’ that would be it.
Ryan Costello
I mean, to all the questions that we had and a major part of the discussion today, I’m going to say AI literacy on that. Knowing the limitations of the tools, knowing the tools that are out there, what they actually do, and how they work. If you can argue effectively for the use of those tools and where there’s potential for cost savings and ROI, but also where the limitations are and how you need to steer and drive that work, that’s going to be incredibly important in the coming years, for sure.
Mary Bennett
Thank you. David?
David Horrigan
I suppose my final takeaway, to be pithy in our remaining one minute, is to quote the old Russian proverb: trust, but verify. Ronald Reagan popularized the old Russian proverb, “trust, but verify.” Whether it’s shepherd citations, key sites, or any other method of documentation, use these great tools, but verify their accuracy.
Mary Bennett
Thank you. Tom?
Tom O’Connor:
I think the point of educating on AI, competency, literacy, or whatever you want to call it, is… Mohammed just asked a great question. How do we know that an AI solution is truly defensible? Are there any guidelines for that? Not yet, but I suspect we’ll see some. And so, keeping up with that and being the person in your firm or organization who has that literacy is going to make you an invaluable asset.
Mary Bennett
I want to thank Tom, David, and Ryan. This was such a great conversation, and thank you all for being so engaged with these questions. It just makes this so valuable. Haystack has a September webcast coming up, continuing the conversation on what’s happening in the industry, what’s current, and what’s next, including trends, technologies, and legal authorities shaping eDiscovery for cyber and investigations. That’s on the 17th. So I thank you all very much, and we’ll see you next time.
Tom O’Connor:
Thanks, everyone.
Expert Panelists
+ Ryan Costello
Senior Managing Director, Advisory Services, HaystackID
Ryan Costello is HaystackID’s Senior Managing Director of Advisory Services. Having worked across diverse locales such as Washington, D.C., Rome, Brussels, and Dublin, Costello is bringing his global perspective to help HaystackID’s clients maneuver this changing terrain, from how to best use emerging new technologies to ensuring their data is secure. Ryan began his career as an independent law professional, working with Project Counsel and Project Counsel Media before taking on Prosearch’s privacy accountabilities as Engagement Director and Head of Data Privacy Services. Ryan earned his JD in English and Journalism at Elon University and his JD from the New England School of Law. He holds the CIPP for the US and the EU, as well as the management certificate, CIPM.
+ David Horrigan
Discovery Counsel & Legal Education Director, Relativity
David Horrigan is Relativity’s discovery counsel and legal education director. The recipient of the 2024 Lifetime Achievement Award from the International Legal Technology Association (ILTA), David is an attorney, award-winning journalist, law school guest lecturer, and former e-discovery industry analyst, having served as reporter and assistant editor at The National Law Journal and analyst and counsel at 451 Research. The author of over 500 articles and reports on legal issues, as well as the annual Data Discovery Legal Year in Review, David is a frequent contributor to Legaltech News.
+ Tom O’Connor
Director, Gulf Coast Legal Technology Center
Tom O’Connor, a nationally known consultant, speaker, and writer in the area of computerized litigation support systems, is a New England native who graduated from The Johns Hopkins University in 1972 with a BA in Political Science. After attending law school for one year at the University of Notre Dame, Tom returned to Baltimore and undertook a career as a paralegal specializing in complex litigation. He later earned a J.D. from an evening law school program. Tom’s consulting experience is primarily in complex litigation matters, where he has worked with firms of all sizes, most recently the Plaintiffs in the BP litigation. He has also been appointed as a technical consultant by various federal and state courts on cases dealing with large amounts of electronic evidence. Tom is a frequent lecturer on the subject of legal technology and is also a prolific writer, with numerous articles in legal publications and several books, most recently the self-published “EDiscovery for the Rest of Us”.
+ Mary Bennett [Moderator]
Senior Director, Content Marketing, HaystackID; Senior Director, Content and Community Initiatives, EDRM
Mary Bennett, HaystackID’s Director of Content Marketing, focuses on the power of storytelling to educate the legal technology industry on pressing issues impacting practitioners. With nearly 10 years of content marketing experience, Bennett joined HaystackID after working at an agency to help B2B tech startups grow their marketing engines through content that drove audiences through the marketing funnel. Before her agency experience, Bennett worked at Chicago-based Relativity as a Senior Producer on the Brand Programs team. She was a founding member, host, and producer of Relativity’s Stellar Women program and producer of the company’s documentary series, On the Merits. In her role, Bennett crafted and socialized important stories that elevated the eDiscovery community and illustrated technology’s potential to make a substantial impact.
Assisted by GAI and LLM technologies.
SOURCE: HaystackID
