Your AI Agent Is Costing You More Than You Think

Ben Lorica

Abi Aryan on LLMOps, Agent Reliability, FinOps, and Reinforcement Learning.


Subscribe: Apple • Spotify • Overcast • Pocket Casts • YouTube •  AntennaPod • Podcast Addict • Amazon •  RSS.

Ben Lorica talks with Abi Aryan, AI engineer, author of LLMOps: Managing Large Language Models in Production, about how deploying generative AI is turning into a distributed systems problem. They discuss the shift from MLOps to LLMOps, agent reliability and observability, why many teams are building their own agent and evaluation infrastructure, the overlooked importance of FinOps and data engineering, and why reinforcement learning environments need to become more dynamic. Abi also explains why AI engineers should spend more time learning system design and model profiling.  [This episode originally aired on Generative AI in the Real World, a podcast series I’m hosting for O’Reilly.]

Subscribe to the Gradient Flow Newsletter

Interview highlights – key sections from the video version:

Jump to transcript

Related content:

Support our work by subscribing to our newsletter📩

Transcript

Below is a polished and edited transcript.

Ben Lorica. All right, so today we have Abi Aryan. She is the author of the O’Reilly book on LLMOps, as well as the founder of Abide AI. So, Abi, welcome to the podcast.

Abi Aryan. Thank you so much, Ben.

Ben Lorica. All right, let’s start with the book, which I confess I just cracked open, LLMOps. People listening to this have probably heard of MLOps. At a high level, I guess the models have changed. They’re bigger, they’re generative, and so on and so forth. Since you’ve written this book, have you seen a wider acceptance of the need for LLMOps?

Abi Aryan. I think more recently there are more infrastructure companies. There was a conference happening recently, and there was this sort of perception, or messaging, across the conference that MLOps is dead. I don’t agree with that, but there’s a big difference that I think companies have started to pick up on more recently as the infrastructure around this space has started to improve.

They’re starting to realize how different the pipelines are from what they were managing earlier, especially for older companies like Snorkel that were in this space for years and years before large language models came in. The way they were handling data pipelines, and even the observability platforms that we’re seeing today, have changed tremendously.

Ben Lorica. And so, Abi, we don’t have to go into specific tools, but we can if you want. If you look at the old MLOps person and then fast-forward, this person is now an LLMOps person. On a day-to-day basis, has their suite of tools changed massively?

Abi Aryan. I think for an MLOps person, the focus was very much around: This is my model. How do I containerize my model? How do I put it in production? That was the entire problem. Most of the work was around: Can I containerize it? What are the best practices around how I arrange my repository? Are we using templates?

Rollbacks happened, but not as much, because most of the time the stuff was tested and there wasn’t too much nondeterministic behavior within the models themselves.

Now that has changed. The biggest job for most LLMOps engineers right now is really doing FinOps, which is controlling the cost of the models. I think the second thing that’s been a big difference is that we have shifted from, “How can we build systems?” to, “How can we build systems that can perform?” And not just perform technically, but perform behaviorally as well.

What is the cost of the model? What does the latency look like? What’s the throughput looking like? How are we managing the memory across different tasks? The problem has really shifted.

A lot of the focus for MLOps was, “Let’s create fantastic dashboards that can do everything.” Right now, no matter which dashboard you create, the monitoring is very, very dynamic.

Ben Lorica. Yeah. And actually, as you were talking there, I started thinking, of course, now inference is essentially a distributed computing problem, right? That was not the case before. Now you have different phases even of the computation during inference. You have the prefill phase and the decode phase, and you might need different setups for those.

So anecdotally, Abi, were the people who were MLOps people able to successfully migrate themselves? Were they able to upskill themselves to become LLMOps engineers?

Abi Aryan. I know a couple of friends who were MLOps engineers. They were teaching MLOps as well, Databricks folks, MVPs, and they were transitioning to LLMOps. But the way they started was by focusing very much on, “Can we do evals for these models?” They weren’t really dealing with the infrastructure side of it yet, and that was their slow transition.

Right now they’re very much at that point where they’re thinking, “Okay, can we make it easy to catch these problems within the model inference itself?” But a lot of other problems still stay unsolved.

Then there’s the other side, which is a lot of software engineers who entered the field and became AI engineers. They have a much easier transition because I look at large language models not just as another machine learning model, but literally as Software 3.0 in that way. It’s an end-to-end system that will run independently. Now the model isn’t just something you plug in, but the model is really the product.

For those people, most of software was built around these ideas: We need strong cohesion. We need low coupling. We need to think about how we’re doing microservices. How does communication happen between the different tools that we’re using? How are we calling our endpoints? How are we securing our endpoints?

Those questions come easier. The system design side of things comes easier to people who worked in traditional software engineering, so the transition has been a little bit easier for them compared to people who were traditionally MLOps engineers.

Ben Lorica. Yeah, and hopefully your book will help some of these MLOps people upskill themselves into this new world.

All right, let’s pivot quickly to agents. Obviously, it’s a buzzword, just like anything in this space. It means different things to different teams. So how do you distinguish agentic systems yourself?

Abi Aryan. There are two words in this space. One is agents. One is agentic workflows. Basically, agents are the components, really, or you can call them the model itself, but they’re trying to figure out what you meant even if you forgot to tell them. That’s the core work of an agent.

The work of a workflow, or an agentic workflow, an agentic system if you want to call it that, is to tell these agents what to actually do. One is responsible for execution. The other is responsible for the planning side of things.

Ben Lorica. Yeah. I think sometimes when tech journalists write about these things, the general public gets the notion that there’s this monolithic model that does everything. But the reality is most teams are moving away from that design, as you described. They have an agent that acts as an orchestrator or planner, and then parcels out the different steps or tasks needed and maybe reassembles them in the end, right?

Abi Aryan. Coming back to your point, it’s now less of a machine learning problem and, again, more like a distributed systems problem because we have multiple agents. Some of these agents will have more load. Let’s say there are front-end agents communicating with a lot of people. Obviously, on the GPUs, these need more distribution.

When it comes to the other agents that may not be used as much, they can be provisioned based on the need and the availability that we have. All of that provisioning is a problem. Communication is a problem. Setting up tests across different tasks within an entire workflow becomes a problem. That’s where a lot of people are trying to implement context engineering, but it’s a very complicated problem to solve.

Ben Lorica. Yeah, and then, Abi, there’s also the problem of compounding reliability. Let’s say, for example, you have an agentic workflow where one agent passes off to another agent and yet another third agent. Each agent may have a certain amount of reliability, but it compounds over time. It compounds across this pipeline, which makes it more challenging.

Abi Aryan. Yep, and that’s where there’s a lot of research work going on in this space. It’s an idea that I’ve talked about in the book as well. I mentioned that, at the point when I was writing the book, especially Chapter 4, in which a lot of these ideas were described, most companies were at a monolithic architecture, but that’s not going to be able to sustain.

As we go toward applications, we have to go toward a microservices architecture. The moment we go toward microservices architecture, there are a lot of problems. One will be the hardware problem. The other is consensus building.

Let’s say you have three different agents spread across three different nodes that are running very differently. One is running on an edge, one is running on something else. How do we achieve consensus if even one of the nodes ends up failing?

That’s an open research area where people are trying to figure out: Can we achieve consensus in agents based on whatever answer the majority is giving? Or how do we really think about it? Should we set up a threshold at which, if it’s beyond this threshold, then this perfectly works?

One of the frameworks that is trying to work in this space is called Masjid. They’re working on the research side of solving this problem in terms of the tooling itself.

Ben Lorica. Yeah. By the way, even back in the microservices days of software architecture, obviously people went overboard too. I think that, as with any of these new things, there’s a bit of trial and error that you have to go through.

The better you can test your systems and have a setup where you can reproduce and try different things, the better off you are, because probably many times your first step designing your system may not be the right one, right?

Abi Aryan. Yeah. I’ll give you two examples of this. Most of the vertical AI companies first tried to use a lot of agentic frameworks. People have used CrewAI, people have used…

Ben Lorica. Not that I hate, but…

Abi Aryan. Yeah. Eventually everyone said, “Yeah, we need to write our own code as well.” And 90% of the people who are working in the space seriously have already made that transition, which is, “We are going to write it ourselves.”

The same happened for evaluation. There were a lot of evaluation tools out there. What they were doing on the surface was literally just tracing, and tracing wasn’t really solving the problem. It was just a beautiful dashboard that doesn’t really serve much purpose, maybe for the business teams, but at least for ML engineers who are supposed to debug these problems and optimize these systems, essentially it wasn’t giving much beyond the error response that we’re getting through everything.

Again, for that one as well, most companies have developed their own evaluation frameworks in-house as of now. The people who are just starting out obviously have not, but most of the companies that started working with large language models in 2023 tried every tool out there through 2024. Right now, more and more people are staying away from the frameworks and LangChain and everything. I think people have understood that most of the frameworks in this space are not super reliable.

Ben Lorica. And also, honestly, a bit bloated, right? They come with too many things that you don’t need, in many ways.

Abi Aryan. Loopholes as well. For example, I reported one of the security loopholes with LangChain and LangSmith back in 2024. Those things obviously get reported by people and get worked on, but the companies aren’t really proactively working on closing those security loopholes.

Ben Lorica. Two open source projects that I like that are not specifically agentic are DSPy and [unclear]. I wanted to give them a shout-out.

So this point I’m about to make, there’s no clear-cut answer. But one thing I noticed, Abi, is that people will do the following: I’m going to take something we do, and I’m going to build agents to do the same thing.

The way we do things is, I’m just making this up, I have a project manager, and then I have a designer. I have role B, role C, and then there are certain emails being exchanged. So the first step people take is, “Let’s replicate not just the roles, but the exchange and communication.”

Sometimes that actually increases the complexity of the design of your system because maybe you don’t need to do it the way the humans do it. Maybe if you go to automation and agents, you don’t have to, in other words, over-anthropomorphize your workflow. What do you think about this observation?

Abi Aryan. A very interesting analogy I’ll give you is that people are trying to replicate intelligence without understanding what intelligence is. The same for consciousness. Everybody wants to replicate and create consciousness without understanding consciousness.

The same is happening with this as well. We are trying to replicate a human workflow without really understanding how humans work.

Ben Lorica. Yeah, and sometimes the humans may not be the most efficient thing. They exchange five emails to arrive at something, right?

Abi Aryan. Humans are never context-defined in a very limiting sense. Even if somebody’s job is to do editing, they’re not just doing editing. They’re looking at the flow. They’re looking for a lot of things that you can’t really define.

Obviously, you can over a period of time, but it needs a lot of observation to understand what that person does, and that skill also depends on who the person is. Different people have different skills as well.

So I would say most of the agentic systems right now are just glorified Zapier routines. That’s the way I look at them right now. IFTTT recipes: if this, then that.

Ben Lorica. Yeah, robotic process automation, I guess, is what people called it, right?

The other thing that I don’t think people understand just from reading the popular tech press is that agents have levels of autonomy. Most teams don’t actually build an agent and unleash it fully autonomously from day one. I guess the analogy would be self-driving cars, where they have different levels of automation.

I think most enterprise AI teams realize that, with agents, you have to treat them that way too, depending on the complexity and the importance of the workflow.

You go first with the human very much involved, and then less and less human involvement over time as you develop confidence in the agent. But I don’t think it’s good practice to just let an agent run wild, especially right now, right?

Abi Aryan. It’s not, because who’s the person answering if the agent goes wrong? That’s a question that has come up often.

This is the work that we’re doing at Abide AI, really, which is trying to create a decision layer on top of the knowledge retrieval layer.

Most of the agents that are built using just large language models, LLMs, I think people need to understand this part: LLMs are fantastic at knowledge retrieval, but they do not know how to make decisions.

If you think agents are independent decision makers and they can figure things out, no, they cannot figure things out. They can look at the database and try to do something. Now, what they do may or may not be what you like, no matter how many rules you define around that.

What we really need to develop is some sort of symbolic language around how these agents are working. It’s more like trying to give them a model of the world around what the cause and effect is with all of these decisions that you’re making. How do we prioritize one decision over another? What was the reasoning behind that?

That entire decision-making reasoning layer has been the missing part.

Ben Lorica. So you brought up the topic of observability. There are two schools of thought here as far as agentic observability.

The first one is, “We don’t need new tools. We have the tools. We just have to apply them to agents.” And then the second, of course, is, “No, this is a new situation. The observability tools have to be more capable because we’re dealing with nondeterministic systems.”

Maybe we need to capture more information along the way, chains of decisions, reasoning, traceability, and so on and so forth.

Where do you fall on this spectrum of “we don’t need new tools” versus “we need new tools”?

Abi Aryan. We don’t need new tools, but we certainly need new frameworks and especially a new way of thinking.

Observability in the MLOps world is fantastic if it’s just about rules. Now people have to stop thinking about observability as just visibility into the system and start thinking of it as an anomaly detection problem.

That’s something I’d written in the book as well. Now it’s no longer about, “Can I see what my token length is?” No, that’s not enough. You have to look for anomalies at every single part of the layer across a lot of metrics.

Ben Lorica. So your position is we can use the existing tools. We may have to log more things.

Abi Aryan. We may have to log more things and then start building even simple, usable ML models to be able to do anomaly detection.

Think of managing any machine, any LLM model, any agent, really like a fraud detection pipeline. Every single time, you’re looking for the simplest signs of fraud, and that can happen across various factors. But we need more logging.

Again, you don’t need external tools for that. You can set up your own loggers as well. Most of the people I know have been setting up their own loggers within their companies.

You can simply use telemetry to define a set of general logs and also be able to define your own custom logs, depending on your agentic pipeline itself. You can define, “This is what it’s trying to do,” log more things across those activities, and then start building small machine learning models to look for what’s going on there.

Ben Lorica. So what is the state of this? Where are we? How many teams are doing this?

Abi Aryan. Very few. Very, very few. Maybe just the top teams, the ones who are doing reinforcement learning training and using RL environments, because that’s where they’re getting their data to do RL.

But people who are not using RL to retrain their models, so to say, aren’t really doing much of this part. They’re still depending very much on external tools.

Ben Lorica. I’ll get back to RL in a second. But one topic you raised when you pointed out the transition from ML to LLMOps was the importance of FinOps, which, for our listeners, is basically managing your cloud computing costs, or in this case, increasingly mastering token economics.

It’s one of these things that I think can bite you. For example, the first time you use Claude Code, you go, “Oh man, this tool is powerful,” and then, boom, you get an email with a bill. “I see. That’s why it’s powerful,” right?

You multiply this across the board to teams that are starting to deploy some of these things, and you see the importance of FinOps.

So where are we, Abi, as far as tooling for FinOps in the age of generative AI, and also the practice of FinOps in the age of generative AI?

Abi Aryan. Less than 5%, maybe even 2%, of the way there.

Ben Lorica. Oh, really? But obviously everyone’s aware of it, right? When you deploy, you become aware.

Abi Aryan. Not enough people, because I think a lot of people just think about FinOps as cloud, basically the cloud cost, and there are different kinds of costs in the cloud itself.

One of the things people are not doing enough is profiling their models properly. Where are the costs really coming from? Are our models compute-bound? Are they taking too much…

Ben Lorica. Are we using reasoning when we don’t need it?

Abi Aryan. Right, exactly. Now, that’s a problem you solve very differently. That’s where, yes, you can do kernel fusion to find your own custom kernels.

Right now, there’s a massive number of people who think we need to rewrite kernels for everything. It’s only going to solve one problem, which is the compute-bound problem. It’s not going to solve the memory-bound problem.

Your data engineering pipelines are what’s going to solve your memory-bound problems, and that’s where most of the focus is missing.

I’ve mentioned it in the book as well: Data engineering is the foundation of first being able to solve the problems, and then we move to the compute-bound problems. Do not start out minimizing the kernels over there.

Then the third part would be the communication-bound problem, which is: How do we make these GPUs talk smarter with each other? How do we figure out agent consensus and all of those problems?

That’s a communication problem, and that would happen when there are different levels of bandwidth that everybody is dealing with: internet bandwidth, the kind of serving speed, different kinds of costing, every kind of transition.

If we’re not really hosting our own infrastructure, then that’s a different problem because it depends on which server and GPU zone you get assigned.

Ben Lorica. Yeah, I want to give a shout-out to Ray. I’m an advisor to Anyscale, but Ray is built for these sorts of pipelines because it can do fine-grained utilization and help you decide between CPU and GPU.

All right, so generally, you don’t think teams are taking token economics seriously. I guess not. How many people have I heard talking about caching, for example? If it’s a query, if it’s a prompt that the LLM has answered before, why do you have to go through it again, right?

Abi Aryan. I think plenty of people have started implementing KV caching, but they don’t really know, again, one of the questions people don’t understand is how much we need to store in memory itself and how much we need to store in the cache. That’s the big memory question.

That’s the one I don’t think people are able to solve. A lot of people are storing too much stuff in the cache that should actually be stored in RAM, in memory.

And there are general applications where people don’t really understand that this agent doesn’t actually need access to the memory. There’s no point. It just slows down the throughput, really.

So I think the problem isn’t really caching. The problem is that differentiation of understanding for people.

Ben Lorica. Yeah, I just threw that out as one element because obviously there are many, many things to mastering token economics.

You brought up reinforcement learning. A few years ago, obviously, people got into, “Let’s do fine-tuning.” Fine-tuning became easy because there became so many services where you can just focus on labeled data, then upload your labeled data, boom, come back from lunch, and you have a fine-tuned model.

But then people realized, “Okay, so I fine-tune, but the model that results isn’t really that great. It’s as good as my fine-tuning data.”

Then obviously RAG and context engineering came into the picture. Now it seems like more people are again talking about reinforcement learning, but in the context of LLMs, and there are a lot of libraries, many of them built on Ray, for example.

But it seems like what’s missing, Abi, is that fine-tuning got to the point where I can sit down a domain expert and say, “Produce labeled data,” and basically the domain expert is a first-class participant in fine-tuning.

As best I can tell, for reinforcement learning, the tools aren’t there yet. The UX hasn’t been figured out in order to bring in domain experts as first-class citizens in the reinforcement learning process, which they need to be because a lot of this stuff really resides in their brains.

Abi Aryan. The big problem here, and very much to the point of what you pointed out, is that the tools aren’t really there.

One very specific thing I can tell you is that most of the reinforcement learning environments that you’re seeing are static environments. Agents are not learning statically. They are learning dynamically.

If your RL environment cannot adapt dynamically, then that’s a problem. Back in 2018, 2019, when things like OpenAI Gym and a lot of reinforcement learning libraries were coming out, there was a line of work called curriculum learning, which is basically adapting your model difficulty to the results themselves.

That can be used in reinforcement learning, but I’m not seeing any practical implementation of using curriculum learning for reinforcement learning environments.

People create these environments. Fantastic. They work well for a little bit of time, and then they become useless.

That’s where even OpenAI and Anthropic are struggling as well. They’ve paid heavily in contracts, year-long contracts, to say, “Can you build this vertical environment? Can you build that vertical environment?” And that works fantastically. But once the model learns on it, then there’s nothing else to learn.

Then you go back to the question: Is this data fresh, really? Is this adapting with the world? It becomes the same RAG problem all over again.

Ben Lorica. So maybe the problem is with RL itself. Maybe we need a different paradigm. It’s just too hard, right?

Let me close by looking to the future. This space is moving so fast that this might be an impossible question to ask, but if you look out, let’s say, six to 18 months, what are some things in the research domain that you think aren’t being talked about enough that might produce enough practical utility that we will start hearing about them in six to 18 months?

Abi Aryan. One is how to profile your machine learning models, the entire systems end-to-end. A lot of people do not understand them as systems, but only as models.

That’s one thing that will make a massive amount of difference. There are a lot of AI engineers today, but we don’t have enough system design engineers.

Ben Lorica. That’s something that Ion Stoica at Sky Computing Lab has been giving keynotes about.

Abi Aryan. The second part is, most likely, I think I’m optimistic about seeing curriculum learning applied to reinforcement learning as well, where our RL environments can adapt in real time. When we train agents on them, they’re dynamically adapting as well.

That’s also some of the work being done by labs like Sakana, which are working in artificial life and all of that stuff, the evolution of any kind of machine learning model that you’re seeing.

The third thing, where I feel like the community is falling behind massively, is on the data engineering side. That’s where we have massive gains to get.

Ben Lorica. On the data engineering side, I’m happy to say that I advise several companies in this space that are completely focused on tools for these new workloads and these new data types.

Last question. For our listeners who are listening, what mindset shift or what skill do they need to pick up in order to position themselves in their careers for the next 18 to 24 months?

Abi Aryan. For anybody who’s an AI engineer, machine learning engineer, LLMOps engineer, or MLOps engineer, first learn how to profile your models.

Start picking up Ray very quickly as a tool just to get started and see how distributed systems work. You can pick any tool. You can pick vLLM if you want, but start understanding distributed systems first.

Once you start understanding those systems, then start looking back into the models themselves.

Ben Lorica. And with that, thank you, Abi.