Your AI Safety Tests Are Lying to You

Ben Lorica

Andrew Burt on Guardrails, Red Teaming, and Why Neither Catches Real AI Risk.

Subscribe: Apple • Spotify • Overcast • Pocket Casts • YouTube •  AntennaPod • Podcast Addict • Amazon •  RSS.

Andrew Burt, CEO and co-founder of Luminos, joins Ben Lorica to unpack Luminos’s new report, A Practical Guide to Agentic and Gen AI Risk Evals. They discuss why single-prompt, low-dimensional AI safety checks miss the risks that matter, why LLM-as-a-judge breaks down once you notice that different models have different “personalities,” and why granular, multi-model evals built around specific sub-risks are what actually let teams ship AI faster and safer.

Subscribe to the Gradient Flow Newsletter

Interview highlights – key sections from the video version:

Jump to transcript

Related content:

Support our work by subscribing to our newsletter📩

Transcript

Below is a polished and edited transcript.

Ben Lorica. All right. Today we have my friend Andrew Burt, CEO and co-founder of Luminos, which you can find at luminos.ai. The tagline on the homepage is “Test Your AI for Risks That Matter in 60 Seconds.” Today, we’re here to talk about a new report they have just released, which I will link to in the episode notes. The title is A Practical Guide to Agentic and Gen AI Risk Evals. Andrew, welcome to the podcast.

Andrew Burt. Thank you for having me. I feel like a friend of the podcast. It’s great to be back.

Ben Lorica. If I were to characterize this report in one word, I guess the word that comes to mind is “holistic.” I’ll try to structure the discussion around the many notions of what holistic means in this report. The first is that simple tests are not enough. In other words, a lot of people test for bias, but what this report seems to suggest is that doing so is insufficient. Andrew, is that a fair characterization?

Andrew Burt. Yeah, and I would say it’s not just insufficient. Everybody loses. We call it high dimensionality. I thought you were going to say high dimensionality, but holistic is better. High dimensionality means having very granular provisions for different risks and pairing those with different models.

Basically, when you use low dimensionality, such as one or two simple prompts in the traditional LLM-as-a-judge approach, you flag the wrong risks and miss the right ones. That means you’re leaving risk on the table while paradoxically slowing things down. Everybody loses. You ship AI models more slowly, and you leave risks on the table.

Ben Lorica. The reason I came up with the word holistic is that you have these dimensions, right? But you also have this notion—is the term you use “sub-risk”?

Andrew Burt. I think so. Internally, we…

Ben Lorica. Within bias, there are actually different risks, right?

Andrew Burt. Yeah, that’s exactly right. When you ask a model, “Is this biased?”—or, to use the example from the report—it’s difficult as a startup to release a report like this because a lot of it is our secret sauce. That being said, we made all these discoveries and wanted to share them.

The example we use in the report relates to deception, which itself encompasses many areas. There’s something called UDAP, unfair or deceptive acts or practices, which is a very large area. If you simply tell a model, “Don’t be deceptive,” “Be honest,” or “Be transparent,” you’re going to get spaghetti-at-the-wall responses. You’ll get flags for risks that aren’t real, while actual risks will be missed.

I feel like I’m not answering your question the way I want to, but if you keep the prompt at a high level, or use only one or two high-level prompts, it just doesn’t work the way it should. The sub-provisions, or sub-risks, are what you actually need. You also need to pair them with the appropriate models.

Then you can get a very granular sense of the risk. One of the nice things about that is that once you understand the granular risk, you can do something about it. It’s much harder to act on, “My model is lying.” It’s much easier to act on, “My model is neglecting to disclose that there is an annual subscription fee for whatever I’m trying to sell.”

Granularity helps in two ways. It catches the right risks, and it allows you to troubleshoot once those risks have been detected.

Ben Lorica. I should actually have started with this, because a crucial component of being holistic, or having many dimensions, is the team. Typically, when people conduct evals, the team is engineering-centric or engineering-heavy. But to identify these different dimensions and subtypes, you need much more variety in the skill sets represented on your eval team, correct?

Andrew Burt. Yeah. Like anything else, you need expertise. In terms of risk, we at Luminos have what we call our legal engineering team. The goal is to determine how we can take legal standards and automate them. The team includes lawyers and data scientists, and it is central to who we are.

That’s exactly what you need. When you look at traditional eval platforms—and we can talk about them in a little bit—you get high-level risks that were built and approached from an engineering perspective. That’s great, and it works better for things such as performance. But when you’re asking how to conduct risk evals, that’s why we say we evaluate “the risks that matter.”

We also say “in 60 seconds,” which I know is a funny thing to say. We can obviously do it much faster, but we find that, especially within the legal and governance communities, evals and the process of analyzing a model for risk take a really long time. There’s no reason they should. If you have an engineering approach to detecting and resolving risks, that process should move very quickly.

The other thing our conversation is focused on today is that we also speak to the technical community. For that audience, we have to use slightly different language. But the core point is that when engineers and data scientists tell lawyers that they’re conducting risk evals, those evals are insufficient. When engineers themselves select tools to detect risk, those tools are also insufficient—not just because they fail to check the box effectively, but because they actually slow you down.

I can talk more about how, but everybody loses when you conduct these low-dimensional evals.

Ben Lorica. Andrew, the typical customers or clients you talk to already have evals in place, right? The concern is that while those evals may cover typical machine learning or engineering risks, such as latency and accuracy, they don’t cover much beyond that. If you try to force your engineering team to conduct a much more holistic or high-dimensional risk assessment, they just can’t do it, right?

Andrew Burt. Yeah. What we usually see—and, again, this is where everyone loses—is legal and governance professionals and data scientists trying to figure out what is sufficient. How do you holistically and reliably measure these models for risk?

Usually, what we hear is, “We’re using safeguards,” which are fine, but they’re wholly insufficient.

Ben Lorica. These are also called guardrails.

Andrew Burt. Yeah, guardrails. They’re optimized for latency, which is fine. They don’t hurt, and they’re necessary, but they’re wholly insufficient for catching the risks you need to catch.

We hear, “We’re using safeguards or guardrails.” Then, “We’re doing red teaming.” That’s really important, but it’s typically meant to simulate adversarial behavior. Most interactions between a model and its users are not actually going to be adversarial. Red teaming is important, but you’re still missing the major risks.

Then we hear, “We conducted some evals.” Usually, that means taking existing performance and accuracy tools and applying them to risk. What we typically see is some variation of LLM-as-a-judge. It’s usually one LLM, and the prompts aren’t written by the right people. Sometimes the system is built internally. Sometimes prompts are being placed into systems designed for performance. But the existing way teams conduct evals is insufficient.

I’ve been saying that I’m going to talk about this, so I’ll add it here, but interrupt me if you’d like. One of the really interesting things we learned is that different models have different characteristics and traits. They really have different personalities.

Claude, for example, tends to be conservative, so it overflags risks. That’s simply part of Claude’s personality. Other models underflag risks. Even when we tried to simulate the basic LLM-as-a-judge approach, we saw different prompts behave very differently.

Even when we placed our high-dimensional prompts into different models, they would sometimes break. That was a really interesting finding. You can build a high-dimensional eval and give it to one model, and it might work. Give it to another model, and it might break entirely.

All of this supports your point that the people architecting and building the foundations for these evals have to be subject-matter experts. They need to build the sub-risks or sub-provisions correctly and understand which models should be tied to which sub-provisions. If you don’t do that correctly, things break. It becomes largely a check-the-box exercise, and you slow down deployment.

Ben Lorica. I have a couple of questions. On the topic of sub-provisions, I imagine they’re important because you want coverage for your tests. But they may also be important to the people deploying these models because they need to understand what to fix. If you don’t flag something at the sub-provision level, you may not know what to fix.

Andrew Burt. Yeah, exactly. Otherwise, you basically receive an alert that gives you a high-level reason your model is causing harm. Then you have a lot of work to do. You have to determine what harm it’s causing and why.

But if you have a sub-provision identifying a specific risk, such as the example I mentioned earlier about hiding a renewal fee, that’s a very specific way of deceiving a consumer. Once you have a flag like that, it’s relatively simple to fix.

The way we conduct evals needs to evolve. We need to move away from saying, “Here’s something really high-level,” and move toward specifics.

I mentioned Claude earlier, but Anthropic pioneered constitutional AI. For those who don’t remember, the initial principles for constitutional AI were the three Hs: be honest, be helpful, and don’t be harmful. I think that’s a quintessential example of low dimensionality and of an engineering-first approach to risk and risk evals that is wholly insufficient.

They’ve since improved. They’ve released a constitution with hundreds of provisions. But even then, if you look at the provisions, you can see major gaps. The subject-matter expertise that we apply through legal engineering is largely missing from the way evals are conducted. That’s why we wrote the white paper.

Ben Lorica. Another thing the white paper points out, as you’ve already alluded to, is that a single model as a judge doesn’t really work. The recommendation is a multi-model approach.

One trend in the AI industry is the realization that frontier models from Anthropic, OpenAI, and DeepMind are great, but they’re big, expensive, and general-purpose. People are now realizing that for very specific, targeted tests, they may be able to use a cheaper open-weight model. It might even be better because they can customize and tune it, and it will be faster because it’s smaller.

Is this the direction you’re going in? We don’t necessarily need these enormous models to conduct risk evaluations across 2,000 languages. I may just need a model that can flag risks in English.

Andrew Burt. Yeah, that’s exactly right. What we are seeing across our customer base with open models is exactly that. It makes the risk landscape more complicated when a company, instead of using a couple of large models, is fine-tuning many different open-weight models for its…

Ben Lorica. Very specific work.

Andrew Burt. Yeah. That creates higher demand for risk evals because it introduces complexity.

In terms of selecting the right models for specific provisions, it’s actually very interesting. Some provisions are extremely granular. It turns out that some perform better with larger models, while for others the reverse is true. It really depends on the provision.

In some areas, you may want something slightly more generic. We’ve found that it can be effective to pair approaches. You may still want to run the more generic prompt, but you’ll also want to run the sub-risks.

This is a very curated process. You have to do it carefully, and different models will perform differently. Ben, this is the art of it.

We get the same question that I think every software company gets: “Why can’t I just vibe-code this?” The answer is that this is the worst thing you could ever vibe-code because you would be relying on a single model and wouldn’t be getting granular. But that applies not only to vibe coding. It also applies to existing low-dimensional approaches to evals and LLM-as-a-judge.

Ben Lorica. My question is this: With the multi-model approach you advocate, a listener may hear these ideas and think, “These are all great. I’m going to try this myself,” which is not recommended, right?

I imagine they would try to build multi-model risk evaluators using open-weight models that they could customize and fine-tune for their specific risk profile. Is that what you end up doing for specific clients?

Andrew Burt. No, we don’t. Well, okay. We have a way to automatically construct the right evals. So, yes, if you…

Ben Lorica. Suppose you work with me, and I’m a company that happens to use Anthropic’s models. I tell you, “Andrew, this is great. We can work with you, but you should use our Anthropic models…”

Andrew Burt. You should use the Anthropic models for the evals?

Ben Lorica. Yeah. So you’re not constrained by that.

Andrew Burt. No, we’re not. If someone really had that request, we could try to figure out how to do it, but it’s a bad idea. Again, every model has its own personality. If you are relying…

Ben Lorica. The point is that you’re engaging Luminos to give you an unvarnished, unbiased assessment, right?

Andrew Burt. Yeah. You’re engaging us to do that and to break down the right risks so that you’re not flagging the wrong ones. When you do receive a risk flag, you know what to do with it.

When we talk to engineers, there’s always an inclination to say, “I can just build this myself,” or, “Why don’t I build this myself?” The conversation is basically, “Okay, here’s the methodology. Go ahead.”

If you want to follow this methodology, you can’t use just one model, so you can’t vibe-code it. You have to find all the right models. Then you have to identify all the right prompts. You need a good high-level understanding of what the prompts should be, and then you need to break them down into sub-provisions.

You have to build a harness that can connect all the different models, ensure that they’re attached to the correct sub-provisions, and make sure you’re writing the sub-provisions correctly. Then you need a system for flagging when something goes wrong, as well as documenting and reporting it.

Congratulations. You just built Luminos.

Ben Lorica. And that’s a one-off. Moving forward, you have to maintain the system. Every time a model gets updated, you have to update your system. You also have to retain all the artifacts in case a regulator calls you.

Andrew Burt. That’s exactly right. The question of whether you should build this internally or use an external provider is straightforward. It makes no sense to build it, which is why we exist.

Right now, we’re really the only ones doing this. If others begin doing it, the question will no longer be whether to use this approach. It will be whether we or someone else does it better.

What you just described is what we call the “day two problem.” It’s something that has existed in risk, governance, and legal work for a long time. You get set up for day one and feel really good. You’ve managed the risks. Then it’s day two, day three, and day four.

You have to think about everything you do not just as a point in time, but across a temporal dimension. You have to ensure that the system is maintained over time, that it continues working, and that it remains up to date.

There is a model component, maintenance of the harness, and all the other technical work. But there is also the substance of the requirements. Legal requirements change, and they vary across jurisdictions. If I’m doing something in China, I may need a set of provisions that I don’t need in the United States.

Lawyers exist because all of this is complicated and sometimes ambiguous. If you want to build something like this internally, it’s going to require a lot of resources.

Ben Lorica. Andrew, suppose I’m listening to this and work in a domain that uses a lot of AI—perhaps social media or something similar—but there aren’t clear regulations. Do I still need holistic or high-dimensional risk evaluation?

Andrew Burt. I’m not going to give a blanket “yes,” even though I’d like to, because the more people who use Luminos, the better it is for me.

The real question is how important the AI system is to your business. The more important it is, the greater the risk. The greater the risk, the more you need good evals.

If your AI system is consumer-facing, if there is no human in the loop, or if you’re using it internally for something sensitive, such as HR or employment-related decisions, then yes, it is a very bad idea to have risk evaluations that aren’t thorough.

Typically, every system needs to go through some level of risk assessment. The major thing we found is that low-dimensional evals flag risks that aren’t actually there. If you care about getting AI out the door, this is a much better approach.

From a purely practical standpoint, doing it this way speeds things up. Even if you feel that the system isn’t particularly sensitive, if it needs a risk review, you shouldn’t conduct that review using a low-dimensional approach. Ultimately, that will slow you down.

Ben Lorica. This is also something you’ve pointed out to me numerous times: At the end of the day, everyone has to worry about reputational risk.

Andrew Burt. Yep, exactly.

Ben Lorica. Reputational risk also demands high-dimensional evals, right?

Andrew Burt. Yeah. If the system is important, it means it’s tied to your reputation. Trust and reputational concerns are huge. If something goes wrong, that is bad for your business.

That’s why there is typically some level of risk and why people use risk evals. They’re aware of it. Again, I think what is new, and what surprised even us, is that there is a practical, nonlegal, nongovernance aspect to this. If you’re doing it incorrectly, it slows you down.

To sum it up, if your AI system has any importance, then if it goes wrong, it can cause reputational harm. That alone is a reason to take this seriously.

Ben Lorica. We’ve established that high-dimensional risk evals are a must. While you could try to build this over a weekend, I don’t recommend it. It falls into the “don’t try this at home” category.

As we’ve discussed, not only do you have to build it correctly once, you have to maintain it on an ongoing basis, especially now that the regulatory landscape is moving so quickly.

Andrew, if I were to engage a company such as Luminos, how long would it take to get this up and running? How much consulting would be involved? And how many—I’m going to invent a job category here—forward-deployed legal engineers would I need?

Andrew Burt. Very good question. You pointed out and made fun of the fact that the website says “in 60 seconds.” That’s there to emphasize that the process is extremely straightforward and easy.

The way it works is that you describe the use case, and we automatically build the relevant eval for you. You don’t need forward-deployed legal engineers.

That being said, we have a legal engineering team for these purposes. Sometimes customers have—I don’t want to say bizarre—very unique needs that aren’t typical. In those cases, we will help them.

That’s also good for us. Through our minimal, forward-engaged legal-engineer approach, we learn from those customers. We can then put those learnings back into the product so that everyone else can conduct these evals in an automated way.

From our perspective, a custom, highly hands-on, forward-deployed approach just doesn’t work. A lot of what we’re doing is solving a math problem involving the volume, variety, and velocity of these models. For anyone who remembers the three Vs from big data 25 years ago, these models simply operate too quickly.

If safeguards aren’t enough and red teaming isn’t enough, you need another approach. You need automated evals, and that’s what we provide.

We’ll help and hold your hand if you need it, but the entire value proposition is that you can get started within 60 seconds.

Ben Lorica. There are evals that I use to decide whether I’m ready to deploy a model, but there are also ongoing evals conducted while the model is in production. Once a model is in production, it may be exposed to new behaviors or data that the initial eval didn’t see. Are you going to continue helping me after I deploy the model?

Andrew Burt. Yeah. One of the things that brings me a lot of joy is that I’ve been doing legal engineering for a while. I stood up the first legal engineering team about 10 years ago, when we were doing data governance at Immuta. The entire point was to bring together risk expertise and technical expertise and marry them.

One of the most enjoyable things about what we’re doing now is that we can actually make engineers happy. Usually, engineers don’t like risk. They don’t want to talk to lawyers, and they don’t want to think about these issues.

But our evals can be built directly into the deployment architecture. If you do it once, there’s no reason not to do it again.

The people concerned about risk can log into our platform and see all the evals being run in real time. It’s all API-based, so the technical teams can simply receive a flag if there’s an issue.

It’s exciting to see engineers happy because they can do this through an API. They don’t have to click around in a new user interface, and the evals can run continuously.

Ben Lorica. This could also help me with incident response.

Andrew Burt. Yeah. You’ll know when there’s an incident as soon as…

Ben Lorica. Because of the notion of sub-provisions, the alert is actually very specific.

Andrew Burt. And you know exactly what’s going wrong.

The frequency with which you run these evals is entirely up to you. If you want to know as something happens, or immediately afterward, you can do that. If you only want daily, weekly, or quarterly snapshots, that’s also possible. It’s highly configurable.

One best practice we advocate is obviously to conduct a risk eval before deployment so that you can be comfortable with the system. You may want to conduct several.

The longer the model is in production, frankly, the more you can trust it. There can still be drift, so you should never simply close your eyes and say, “Great, it’s deployed.”

You want to stagger—or, more accurately, front-load—the evals. During the initial deployment, you conduct them at a high frequency so that you can detect any surprises. Then you can move to something a little more segmented.

Ben Lorica. To learn more, listeners can go to luminos.ai. I’ll also link to the white paper in the episode notes.

One quick question before we end: How often do open-weight models come up, and how often are you seeing them relative to proprietary models?

Andrew Burt. Good question. Things change so quickly. I’ve been doing this for a long time, and I’ve been astounded by how much agentic AI has changed everything over the last three months. The conversations are different, and the substance is different.

What we’re beginning to see is that companies deploying AI more seriously and across more use cases are moving toward open-weight models for the reasons you discussed at the beginning of the show. They’re cheaper, faster, and make it easier to use more models.

That being said, we see many customers with fewer use cases, and I think it’s easier for them to rely on some of the heavier models.

The trend is toward open-weight models, but that doesn’t mean people aren’t still relying heavily—or overrelying—on the larger models.

Ben Lorica. OpenAI and Anthropic, go public as soon as possible, before people realize that price pressure is going to eat your lunch. And with that, thank you, Andrew.

Andrew Burt. Thanks for having me.