Defensible AI for eDiscovery with Aaron Pribil and Todd Haley

Russ Fordyce 🏆 (00:01.203) Welcome back to the Winter Circle, everybody. I'm here today with Todd and Aaron from Haystack ID, and they are tackling the big problem of e-discovery in a modern world with AI and how we're going to tackle these new challenges of all that documentation and all the regulatory concerns about it. They've just won an AI Excellence Award. So gentlemen, congratulations on the award, and welcome to the Winter Circle. Aaron Pribil (00:23.587) Thank you. Todd Haley (00:23.658) Thank you. Russ Fordyce 🏆 (00:25.171) So Todd, why don't you give us a little bit of background on Haystack ID and kind of what you guys are up to. Todd Haley (00:32.63) Sure, absolutely. Haystack ID is a data services and products company. We've been in business since 2011 and really have taken the lead on customized solutions for corporate, law firm, and government clients. The goal of our services is really big enough to matter, small enough to care. And the idea behind that is that we are trying to provide white glove service. We are a services first company. We provide products and have built and included products in those solutions in order to ensure that we bring services to our clients that are needed. Obviously, based on what we're talking about today, one of the biggest new services that we're offering is Genitive AI and Adjunctive AI in a very robust way. And so we're looking to apply that AI technology against that services level that we've always offered. Russ Fordyce 🏆 (01:26.077) Yeah, and a lot of companies are experimenting with AI and they're building into their platform. But, you know, in my experience, very few are going at it the way you guys are in these high risk environments. When you talk to legal teams and government agencies, what's the biggest concern they have with AI? Todd Haley (01:43.508) I think the biggest answer I would say there as a concern is making sure that there's validation around the AI. Obviously, with the concerns around hallucinations, around the issues around AI, people do have those concerns, though really because of the closed systems we use and the testing that we've done. Haystack ID has been doing testing and POCs around AI for over five years, right? Looking at different technology. investigating how it works and ensuring that we use proven court tested validation models against the results to ensure that it works and what we're really finding in all of our testing is that our AI has upwards of 90 % and above recall and in the 70, 80, high 80s for precision, which is really insane compared to what we've been doing for many, many years. So while there is some concern and the concerns that individuals are being brought up, we're completely aware of those concerns, which is why we use other methodologies that we've been using for years to ensure that we validate the results and are happy with it. But I will tell you that AI is absolutely part of our mission and our information today. Russ Fordyce 🏆 (02:58.589) Well, in any discovery, you're dealing with massive amounts of sensitive data. These are court documents, financial records, know, whatever, health records, whatever. What makes this space particularly challenging for a Todd Haley (03:11.796) I think it's being careful about the type of AI that you use. We all use AI as if it means a single element or a single idea, but it really isn't. We are very much using closed, secured systems that meet our SAS and ISO requirements around our security features. And we're ensuring those systems only use the data by which we are looking at and no public data or public information as well. So ensuring the security of the AI is absolutely essential to what we do. Russ Fordyce 🏆 (03:42.687) Yeah, it sounds like a lot of guardrails and a lot of kind of rules for kind of what it can look at, which is it seems to be a big problem. I know in my own experience, you ask it a question and then even in a different thread, you'll get an answer from somewhere else you didn't really expect. Why do most AI tools struggle in environments like this for accuracy? Todd Haley (04:04.27) I think I want to challenge the premise of your question a little bit. think when you're talking about open AI systems, a lot of what you were talking about is right. But in the closed systems that are designed for purpose, the AI does not have the same challenges that you're talking about. It is very organized, structured, works within a particular LLM, and ensures that the data is working with the data that you're giving it. And so we just have to be careful. Like I said, when we talk about AI, I find it's like what processing used to be in the old days. That word means a lot to a lot of people. But what we're really talking about is secured, proven systems to be able to provide the kind of capabilities in AI for our industry. Aaron Pribil (04:51.724) Yeah, Russell, would think to add additional distinction to that, think a lot of the perception is that when people out there using AI, the difference of what you were describing as validation or if it's hallucinating is what is that data grounded in? When you're using these large open models, it's the whole world's worth of information in there. Russ Fordyce 🏆 (04:52.02) Now. Aaron Pribil (05:18.282) you have to be very succinct at what you're prompting to get the data. But even getting that data has to be validated and looked at. And that's just kind of in the broader enterprise, like outside of the legal world, you know, use of AI for us, AI in use, especially in a legal regulatory environment that we live in is very scrutinized. We look at this extraordinary close. We have to have the data grounded. very specifically around any given kind of legality to ensure that we're getting the most accurate and correct information. Russ Fordyce 🏆 (05:57.811) So I know this main platform you're talking about is called CoreFlex. And I think you guys describe it as a command and control system. What does that actually mean? Aaron Pribil (06:08.942) So CoreFlex for Haystack is, think of it simply as the front door for our clients to enter. It's allowing our clients to come in, manage their matters, manage their user, manage their data, leverage all the services that Haystack offers from advisory, forensics, and what we're talking about here today specifically around AI. And to what Todd mentioned, we are very focused on client service and white glove service. So when the clients are coming in and they have this means to come through a front door and self-serve, it's not self-serve unguided. We're there every step of the way helping them. So when they have a particular matter that they load data to us and they say, hey, we need to use AI on this, they can connect with us. We can help. drive the data through the system with them every step of the way and helping them leverage our AI solutions. Russ Fordyce 🏆 (07:12.713) Well, I was reading through your nomination and one of the things that stood out to me was that you guys spent a lot of time on the intake process. Why did you decide to do that? You're of spending so much time on this intake process and what's impact on quality? Aaron Pribil (07:30.532) I think the importance around that is, you know, guiding the clients to using, you know, what we see as best practices. So in that intake process, making sure that they're setting up their matters appropriately, that they're getting this appropriate data that needs to come into the ecosystem of CoreFlex. And then from an additional, like intake and guidance perspective, we actually break down what is the information that we need categorically to help build out. AI solutions for them, whether it's for privilege, reverent, help me on the word here, I'm struggling, relevance, thank you, I'm tired there. And getting this information together, initially from their perspective, any documents that pertain to the matter that are important and over to our LDI team, which is Legal Data Intelligence team, that then can take... Todd Haley (08:09.013) Relevance. Aaron Pribil (08:28.055) review that information, get with the client, give them feedback, consultative advice on how to form up their categories before it gets submitted into our LLM and basically start to do all the analysis that the client's looking for, feedback about what is my matter, about what are the important documents, where are the documents I don't care about. maybe they find something in there that they weren't even aware of that gets flagged. So it's a very robust and a very guided system to make it as flexible, hence the core flex, flexible for our client as much as possible. Russ Fordyce 🏆 (09:09.907) Well, in a lot of these AI tools, and I'm sure it happens in the e-discovery business too, they're talking about how speed and it's easy to use. You guys are really prioritizing the governance. Why do you think that's a better long-term approach? Todd Haley (09:25.11) I guess I'll take a shot at that, Aaron, and then I'll let you fill in. To us, remember, speed has always been important in the discovery industry. But I can tell you without a doubt that if our clients thought the only thing we cared about was speed, we wouldn't have the business that we do today. In our argument, quality must come first. And so when it comes with AI, AI governance and AI intake procedures, AI understanding of the data that's being input, garbage in, garbage out, you have to be careful of. We have to have that type of high end quality first and foremost, and then get the speed. The beauty of it is we do pretty good at getting them both. But the idea is that we always want to make sure quality is first. Russ Fordyce 🏆 (10:14.845) Aaron, anything to add there? Aaron Pribil (10:17.195) I think that pretty much hit everything on those avenues. We just want to make sure that what the clients are getting service-wise from us are what they expect. And additionally, for the information that they're wanting to get out of their matters and the AI component of what they're using with us, that we guide it and have it be a guided process. The last thing we want them to do is submit something over to us and it results in a massive bill versus if we're there and we're consultative in helping them, guide them through that experience and kind of trimming away any information that doesn't need to be there, it just makes a better experience for the client and for us as well. Because if our client's happy, we're happy. Russ Fordyce 🏆 (11:11.251) Well, and in this regulated environment that you guys are working in, you've definitely built in this kind of human in the loop experience. And Todd, mentioned it, or excuse me, Aaron, you mentioned it, kind of that legal data intelligence team. What role are they playing in your workflow? Aaron Pribil (11:27.695) So they're very much our subject matter experts. They're there to help guide the client and AI, know, for most it's relatively new, not a new thing, but new overall in the last couple of years. It's really taken off a lot of, you know, hype around it and everybody looking to see how AI can solve, you know, their problems. But unconsulted or on left to, you know, not having experts in the loop on that, you know, can result into, you know, some pretty disastrous things. There's been, you know, lawyers out there that have, you know, used AI but didn't validate, check what the information was and that causes sanctions and other things. We, you know, we want to avoid those things at all costs. And that's, know, what our LDI team is there to do to help make the, you know, client informed and a guided process. Todd Haley (12:28.278) Yeah, I'll give you a quote from one of our clients on that. One of the clients that did a very large project with us said that they had done AI projects with other industry leaders. And one of the things that they found about our process is that the consultative component of our AI solution is what actually made a significant difference in their overall. Russ Fordyce 🏆 (12:28.756) See? Todd Haley (12:50.088) review, their privilege analysis, and the amount of money that they save because the way we do sampling, the way we determine how the AI should be used is through a really consultative understanding of talking with the client, understanding what the matter is, what they want to do to win, and what information they're looking for. And then we tailor a very customized consultative solution around that. Russ Fordyce 🏆 (13:16.071) Yeah, and one of the biggest kind of issues that most companies have, especially in these regulated environments, is it's that black box problem, right? You don't kind of know what's going on in the background. How do you guys address that? Todd Haley (13:30.806) Well, look, we work with some of the largest banks in the world, right? And we're actually working right now on a major project with one of those banks around with AI and an AI solution. And that can't be more regulated than any other industry. There are tons of regulations around that. And again, it's only a black box in the sense that there's not an understanding of it. There's no black box. There's a mathematician that completely understands the algorithm of how LLM works. Our job is to take that mathematical precision and understanding and convert it into layman's terms to make sure that we're covering everything about it. So when we talked earlier about intake, about how we manage the data, how we govern it, how we validate against the results, all of that eliminates that black box problem. Because at the end of the day, we understand what we put in, we understand what came out, and we know why what came out ended up being there. So there is no black box. The black box is just something we answer because we may not understand the algorithm within it, but there's no black box. So I can give you our LDI team who can very much talk about the algorithm and how it works. Most people don't care as long as they get the results and have validation that those results work. That's what they're most interested in. Russ Fordyce 🏆 (14:51.219) Well, now the impact of this has got to be enormous. My father was an attorney and he spent decades in lobbying Congress and going through government documents and all sorts of stuff. And it was days, hours, weeks, just a tremendous amount of time to even get an inkling of what was going on from the big picture. I imagine you guys reduced that down to seconds or hours for these legal teams. Aaron Pribil (15:21.719) Yeah, it is really not that fast. The only limitation is how big is the pipeline to get that data to us and how much resources we got to chew through that data and analyze it at the end of the day. It's night and day difference to what it used to be just four years ago. Todd Haley (15:41.814) Yeah, I will caveat. So yes, absolutely. We're looking at 60 % to 70 % savings in time, cost, et cetera. But I also want to caution individuals that it's not an easy button. So one of the things that my dad, having been a carpenter, always said, measure twice, cut once. One of the challenges I find when watching some of the people using AI right now is they're cutting and then realizing they measured wrong. And so part of it is that it might take us four extra days to prep this work, to get it right, to understand what we're gonna do. And sometimes we want us to rush that process. Well, the reality is if we take a few days upfront to organize and plan, the backend benefits are exponentially greater than any other thing that we've ever used in our industry. Russ Fordyce 🏆 (16:34.367) Well, and you mentioned that 60 % number and I imagine your customers and the legal teams working on these documents are looking for ROI all over. So is it really cost savings? Is it time? Is it risk? Todd Haley (16:48.66) Yes, all the above, right? I go ahead. I mean, go ahead, Aaron. Aaron Pribil (16:49.764) think it's smaller. Yeah, smaller than both. Well, I'll just go ahead with his hat. Todd Haley (16:57.076) Yeah, so no, I was going to say that really what I look at it as is, yes, let's be fair. When people talk about time and cost in the legal industry, they're equally combined, right? The idea is that if we save you time, which is what our first and foremost thing is, by the nature of what we do, we automatically save you costs. And so the idea here is to get the highest quality result in the shortest amount of time and ensure that the data that they're then looking at is the most responsive to what they're looking to achieve. We can do that really, really well. And we do it in all areas. We do it at the front end with early case assessment. We do it as part of relevance and we do it as part of privilege and even case strategy today. And so the idea here is we wanna continue to do that and make sure that we're doing it in a way that makes perfect sense. Russ Fordyce 🏆 (17:54.311) What surprised you most about how quickly the clients are adopting this? Aaron Pribil (17:59.523) I, for me, it, you know, being in this industry, I think you guys have me both beat. I've been in this industry only for about 20 years, but just overall general premise of technology adoption in our space has always been slow and painful. It's, it's unbelievable how quickly AI has flipped the script on that. And everybody is rushing to figure out how do we institute this, this new technology. And if we don't. we're not staying competitive. And I think that's the pushing factor is that our, know, within the law world and arena see how much it's upheaving every other arena. And if they're not using it, they could be left in the dust. I think that's kind of what people look at, but they also have to look at it with, you know, risk reduction and making sure that it's being. approached carefully. So I think the one nuance or difference, I kind of mentioned this at the beginning, is that the other sectors of a business that are using AI are like, let's figure out how we make this, monetize it, let's get this going, but they don't look at what are the legal implications? What are the challenges of if something goes sideways and we got a legality? what kind of hot water we're in, where for us in the legal arena and using AI to help people in their cases or when I say people in corporation law firms, we're looking at that very razor sharp angle of how do we leverage AI and how we do it in a defensible manner and do it in a way that is reducing risk and not opening up. I'll stop because I'm... rambling here a bit, but the one thing that I saw recently like was with a a lawyer's client that was using chat GPT to get legal advice. And then that ended up being discoverable in court. And it wasn't attorney privileged anymore because you're, you're interfacing and using AI versus talking with your, your lawyer directly. You know, it's those things that people don't think about that with us and our legal hats on are always very mindful of what we're doing. Aaron Pribil (20:21.946) with AI. Russ Fordyce 🏆 (20:23.827) Yeah. Do you think most companies are approaching AI the wrong way today, or do think they're going at it the right Todd Haley (20:31.01) I think that's loaded question, right? Because what defines right? So we have to be really careful depending on what we're looking for, what they're trying to achieve, how they want to do it, and then go about it. I do think that companies are looking at AI appropriately in the way that it is a tool, just like anything else we've ever used. I mean, know that all the movies have scared us about AI. Aaron Pribil (20:31.032) I think they're. I think they tried, you know. Russ Fordyce 🏆 (20:39.045) It works. Todd Haley (21:00.116) But the reality of it is that it's no different than any other tool we've ever come up with. We have to have the guardrails. We have to have the governance. We have to have the understanding to manage that tool in a way that makes sense. If you allow the tool to do your job for you, then you're doing it wrong. It is a tool. It is not a replacement of you. Right. And so making sure that you slow down, you use all the legal and data knowledge that you've gained and your peers have gained and use that to be able to get to that answer faster, I think we're in good shape and I think that's what most people are trying to do. Russ Fordyce 🏆 (21:39.805) Yeah, I look at the landscape of AI and I see all these kind of specialists kind of training data. You guys have this kind of more structured AI system, a closed system. And I liken it to tennis rackets or fishing rods. You can go down to Walmart and buy a fishing rod or a tennis racket, but you're not going to win Wimbledon or the Bass Pro Tournament with a fishing rod from Walmart. These specialized systems are, I think, are really gaining traction. Do you think we'll see a big shift from these kind of open-ended AI tools into more structured AI systems like yours? Todd Haley (22:18.164) I think it's both. think the answer is that open AI is going to be there for the general population. I think that is how our kids still are going to learn to some extent. I think some of our adults in the regular world, the personal world, will use open AI. I think when you get to the professional levels though, I think you're going to find that individuals are using structured, secured AI solutions and that are targeted for purpose. Right. Because one of the things you want to be careful when you're looking at the large language models, each of them does something different. mean, I always talk about chat GBT today. If you look in chat, TV for when it first came out, there's multiple ways you can hit their models by focusing on images, focusing on audio, video, et cetera. That open now imagine that in a really tighter situation, which structured closed LLMs where they are targeting for a purpose. to achieve an absolute goal. think those types of systems are gonna be used more and more when you're at the professional level, ensuring defensibility, ensuring the legality, and even outside of the legal realm within a corporate data or technology company, for example, they wanna make sure that the data they're responding to is specific to their information and their systems. So keeping it proprietary as well. Russ Fordyce 🏆 (23:42.239) Well, you mentioned defensibility. Talk us through how the CoreFlex platform has that defensibility. Todd Haley (23:48.744) So I think the way you want to think about it is CoreFlex is really an abstraction layer. Our portal in is actually tying directly to already proven and secured technologies. takes the ability of a window into our soul and really focuses on tools that have already been core proven, have already been used in matters. Even our AI has been already approved by most of the regulatory bodies, at least in the United States, and we're working on the EU. Aaron Pribil (23:48.912) So. Todd Haley (24:18.588) where we're gonna be able to do all of that stuff. We have to be able to have defensibility. We also have to meet regulations. We have the EU Act, AI Act that's now in place that we are very much in compliance with. We work with all the US security things. The idea is figure out the defensibility, figure out the regulatory requirements and ensure that you're within those walls. Russ Fordyce 🏆 (24:45.607) And if somebody's out there listening and they're thinking about building this type of platform and going into this area of the closed model or this kind of closed system, what kind of lessons did you learn and what would you guide them to do differently? Aaron Pribil (25:05.647) Well, I would say it's there's no magic behind how you, you do this. I think first and foremost, it's, listening to the market and to your clients. You need to serve something that clients are willing to use and there's a need for, that that's, you know, first and foremost, the biggest thing then the challenge they're after is, is stack ranking. What do I work on next? What do I look at next? And balancing the market needs and the active clients and what's going on in the marketplace that our challenges that need to be solved that our clients are facing. Russ Fordyce 🏆 (25:50.344) and ta- Todd Haley (25:50.452) Yeah, I'll add to that. We actually met with a corporate client last week on a consultative call, which I think is valid here, which is when we talked to them about their AI initiatives internally, they had over 40 different AI initiatives. And our first response was, let's pick the top five. Let's focus on those. Let's work with those, analyze your work product, make sure that you're using an AI as a tool, not as the answer. and then make sure from those five, get those right in 2026. When you get those right, use all the lessons learned from your experience in order to do the other 30 some initiatives, right? HACS learned from its experience. You heard me say we've been in this for years. We didn't roll it out though until we were ensured that we had done the first POC ever done, that we had separate validation methods that we had used. and we were confident with the results before we put it in front of our clients. Russ Fordyce 🏆 (26:52.415) And what can we look for for Haystack coming up in the next year or two? Aaron Pribil (26:58.865) Um, quite a bit. We're, uh, very heavily focused on, you know, I hate to say use the buzzword AI, but that's the theme of, you know, what we're out here, but very focused on, on further, you know, AI integrations into our platform, as well as making it, uh, easier for our, uh, clients to, you know, bring data in, uh, in through a self-service manner. Um, but even outside of, of, of that, um, you know, we're also looking at. how do we move more left and in terms of left meaning the origins of where the data started, how can we help our clients mitigate risk, prepare for something that's incoming and get it to the end game of if something happens legality wise, we're able to get them from A to Z. Russ Fordyce 🏆 (27:51.109) Aaron Todd, this has been super interesting. can't imagine ever going back to where we were. You think about how far we've come in this. I can't imagine how they did it before. It was armies of people and years and years. And now we're talking about actual efficiency. So congratulations on the award and congratulations on all the great work. Todd Haley (28:15.818) Thank you. Aaron Pribil (28:16.705) Thank you, Russ.