An Agent Is Just an LLM in a For-Loop

Ben Lorica

Maarten Grootendorst on Demystifying AI Agents: What Works, What Fails, and What’s Just Hype.

Subscribe: Apple • Spotify • Overcast • Pocket Casts • YouTube •  AntennaPod • Podcast Addict • Amazon •  RSS.

Maarten Grootendorst, a developer relations engineer at Google DeepMind and co-author of Hands-On Large Language Models and An Illustrated Guide to AI Agents, joins Ben Lorica to explain the concepts developers need to understand beneath today’s AI tools. They discuss why BERTopic and embeddings remain valuable, what makes a system truly agentic, where coding agents work—and fail—and why open-weight models matter for privacy, control, and efficiency. The conversation closes with perspectives on reinforcement learning, changing scaling laws, personal-agent security, and the growing tension between AI demand and data-center infrastructure.  [This episode originally aired on Generative AI in the Real World, a podcast series I’m hosting for O’Reilly.]

Subscribe to the Gradient Flow Newsletter

Interview highlights – key sections from the video version:

Jump to transcript

Related content:

Support our work by subscribing to our newsletter📩

Transcript

Below is a polished and edited transcript.

Ben Lorica: All right, so today we have Maarten Grootendorst. He is a developer relations engineer at Google DeepMind, and he is also the co-author of two O’Reilly books: Hands-On Large Language Models and An Illustrated Guide to AI Agents. So Maarten, welcome to the podcast.

Maarten Grootendorst: Thank you. It’s wonderful to be here.

Ben Lorica: I had you on a podcast — I was looking at it earlier this morning — back in August 2022, a few months before ChatGPT was released. Back then, I wanted to talk to you about BERTopic, your library. For listeners who are not familiar, BERTopic was a marriage between the transformer approach and topic modeling, and Maarten wrote one of the more popular libraries for doing that. So what’s happened to topic modeling since then?

Maarten Grootendorst: I think it’s still going strong. A lot of people say, “Just use ChatGPT for topic modeling,” and you can, but it’s very difficult to get structured, standardized output — especially if you’re dealing with millions of documents. BERTopic is still my baby, in a sense. It’s been four years since we talked, and I love working on it. I don’t have as much time as I used to, but it’s great.

Ben Lorica: I think one of the things large language models have done is cast aside some of these earlier approaches for working through large amounts of text. People are trying to prompt their way into a topic model, but topic models themselves are still very useful. So what’s the level of usage of BERTopic now compared to when we last spoke?

Maarten Grootendorst: It’s only grown since then.

Ben Lorica: Really? Yeah.

Maarten Grootendorst: It surprised me too. I think it’s because it’s easy to use. The main benefit was mostly just a nice user experience, and that helps people accomplish a very specific task instead of prompting their way toward something that may or may not work. It just works out of the box. It’s not perfect — nothing is — but yeah, I think that’s it.

Ben Lorica: One thing that’s happened, of course, is that AI and NLP have become so democratized. When we talked, I think the people using BERTopic at least had some notion of what NLP and text mining were. I would imagine now, in your role in developer relations, you encounter a lot of people who don’t come from a data science or ML background and have no idea what topic models are.

Maarten Grootendorst: Yeah, many don’t. It’s very interesting, because you mention NLP and text mining — completely outdated terms now, for some reason. It’s all just “AI.” That’s not necessarily a bad thing, it’s just interesting to see how the field has evolved. But it also means people don’t look toward these older techniques that still drive much of the adoption of newer tools. Sometimes it feels like AI and LLMs are a hammer, and we’re looking for nails to use it on, instead of recognizing that we have packages built for very specific things, and you can use LLMs on top of those — but you don’t have to. It requires a bit of education, because a lot of people new to the field need to have things like embeddings and clustering explained to them. It’s a nice opportunity for me to explain things, and I enjoy doing that.

Ben Lorica: The key issue is that because so many people are entering this field and building things without knowing the prior art, they might be leaving a lot on the table. They have their text or data, they prompt, and they think they’ve gotten everything out of it — but that’s often not the case.

Maarten Grootendorst: Definitely not. There’s so much you can do with these systems, whether on the LLM side, the agentic side, or the topic modeling side. If you know a little more about what’s going on under the hood, it helps you understand when to prompt, when not to, and what’s going wrong. That intuition is much harder to develop if you don’t really understand how an LLM works.

Ben Lorica: Which brings me to your two books, which are fantastic and go a long way toward helping people build that foundation. But let’s face it — a lot of people don’t have time to read an entire book. For a developer without a data science or ML background, what are the three or four most important concepts about large language models that would set them up for success?

Maarten Grootendorst: Off the top of my head, those are covered in chapters two and three — so buy the book. No, I’m just kidding. The first is tokens. Tokens are super underappreciated.

Ben Lorica: Which is now a big topic, because — as I joke — the CFO has become the CTO: the Chief Token Officer.

Maarten Grootendorst: I didn’t know that one. That’s amazing — I’m going to use that. But yes, tokens are the thing now. They’re how LLMs see and interpret the world, and how they communicate with it. Understanding tokens leads you naturally into embeddings, which I still think are fundamental to so much of what we do. The second is the attention mechanism — why are these models so powerful? What makes them special? Attention is the obvious answer. We have other approaches like Mamba and recurrent neural networks, but it all starts from attention. If you’re completely new to the field, those two topics are where I’d start.

Ben Lorica: Let’s take embeddings. At least some people have had to play around with them, because when LLMs first came online, the “Hello World” example was RAG. And in RAG, one of the knobs people were turning was embeddings — along with chunking, information extraction, and search and retrieval. People could go to Hugging Face and try four different embedding models. Do you find that more people have at least a rudimentary understanding of embeddings because of that?

Maarten Grootendorst: Yeah. I have a soft spot for embeddings because they’re the core of BERTopic, but I think they’re fundamental to so many things in this field. Even RAG — which some people think is outdated, but really isn’t — runs on embeddings. Understanding how they work helps you understand how LLMs work, and they can be used in so many different ways. Sometimes we want larger, contextualized embedding models. Other times, there’s value in static embeddings — super fast, like the old-school embeddings we used to have, now in a new form — which can be used with a coding agent to quickly search through repos. Much of what we do is still search, and search relies heavily on embeddings. Having one numerical representation for a piece of text opens up so many possibilities.

Ben Lorica: Is there a canonical example you use to convince someone that they should learn more about embeddings?

Maarten Grootendorst: I love that question, but I don’t think I have one single answer — it depends on the application. That said, a nice example is the difference between contextualized and non-contextualized embeddings. Contextualized ones are generally larger transformer-like models that require a lot of compute, so you can actually see latency appearing in your search engine, or your coding agent slows down waiting for results. Compare that to faster static models like Model2Vec, which are tremendously fast — great for those use cases, though lower in performance because they’re much smaller. These are the kinds of use cases where hands-on building gives you a lot of intuition about when to use what, rather than delegating that decision entirely to an agent.

Ben Lorica: Though realistically, people will just go to a leaderboard, right? They’ll find the top ten, notice some are larger than others, and try three or four of varying sizes.

Maarten Grootendorst: Yeah, that’s what even I always did — just pick one or two from the top of the leaderboard. But as you gain experience, you start asking: what about multilinguality? I’m Dutch, and there aren’t many strong Dutch embedding models, which is a real problem. There are things like Matryoshka embeddings, where a single model generates embeddings of different sizes for different purposes. And now we have instruction-tuned embeddings, where you prefix with an instruction — for clustering, for classification, whatever — and suddenly you see all these nuances in the selection process.

Ben Lorica: On the attention mechanism — I’ll play the role of someone with no time. What are one to three things I should know about it?

Maarten Grootendorst: The most important thing is that it contextualizes information. Before attention and after attention — it’s not completely black and white, but that’s the key shift. It puts things in context. If you have the word “bank,” is it a riverbank or a financial bank? And as we’re talking right now, a lot of contextual information is needed to interpret what I’m saying. If you only focus on my words in isolation, you lose the fact that there was a question beforehand driving my answer. Attention tries to look at the entire context rather than individual tokens or words in isolation.

Ben Lorica: Playing devil’s advocate — you just explained it to me. Why do I need to learn more than that?

Maarten Grootendorst: Always learn more.

Ben Lorica: You mentioned Mamba and state space models. There was some excitement around them. Can you give our listeners a high-level description of what state space models are and what their current status is in practical usage?

Maarten Grootendorst: State space models are a completely different approach to the attention mechanism — they almost do away with it and replace it with something much faster. It’s a very complex and technical subject, so I don’t want to go too deep. But essentially, what you see happening is that people replace some attention layers in a decoder-based LLM with state space model layers. Normally, an LLM has several stacks of attention mechanisms. You can replace some of them with these faster state space models to speed up inference, because inference speed is what we’re mostly constrained by now. People want more tokens, so it needs to be faster.

Ben Lorica: And what’s the actual adoption of state space models right now?

Maarten Grootendorst: Mostly hybrid models — models that interleave attention blocks with Mamba blocks to speed things up. Some approaches use local attention and global attention, where one is more compute-intensive than the other. Mamba is a way to do something similar: speed up inference by replacing some of the heavier attention layers.

Ben Lorica: Your latest book is about agents — An Illustrated Guide to AI Agents. Before we dive in, what makes a system truly agentic? Before the word “agents” became ubiquitous, people were using terms like robotic process automation. In your mind, what makes a system agentic?

Maarten Grootendorst: That’s actually been one of the more complex topics for us to define, because the field changes so quickly. What fundamentally is an agent when the definition shifts every two months? It’s a bit of a hot take, but I really do think an agent is an LLM in a for-loop with some tools, some memory, and perhaps some guardrails. When you boil it all down, that’s essentially it.

Ben Lorica: So you just described the harness. And “harness engineering” is also a hot term right now.

Maarten Grootendorst: Yep.

Ben Lorica: So where is the real progress with agents, and what is just marketing?

Maarten Grootendorst: Agents sound cool, and they are cool, but the moment you give an LLM complete freedom — no constraints, just go do your thing — it will fail horribly. Agents still need direction; they need to be constrained in what they do. There’s a lot of hype, and I’m not a big fan of hype, but there are genuinely cool use cases. Coding agents are the big thing right now for good reason — I use them daily, they make my life easier. But for other use cases, we’re very early. Coding works nicely, but asking an agent to book a vacation across five countries? No, not yet.

Ben Lorica: That example — plan a trip across five countries, pick the best hotel and flight for each — was always the demo even during the robotic process automation era, and as you said, we still can’t quite do it. What about deep research agents? Would you consider deep research an agent?

Maarten Grootendorst: Oh, definitely — though it depends on the implementation. You can make it very structured: search on arXiv, read the abstract, make a summary, done. That’s not really an agent.

Ben Lorica: That fits your description though — you prompt an LLM, it goes into a for-loop, uses tools like a search index or knowledge graph.

Maarten Grootendorst: Fair enough. The key distinction is whether the system makes its own decisions about when and why to use a tool, versus being put in a pipeline where you explicitly say “always do steps one, two, and three.” An agent might decide to do steps 3, 3, 1, 2, 1, 3 — deciding on its own when and where to use specific tools. I think that’s maybe the best distinction between what is and isn’t an agent.

Ben Lorica: Memory could also play a role there — especially if I’m using one service consistently, it might remember my preferences over time.

Maarten Grootendorst: How we phrase it in the book is: an agent is a reasoning LLM that has access to planning, tools, and memory. There’s no such thing as an agent that completes three steps only to forget what the previous steps were. Memory is underappreciated in the realm of agents. Imagine having to go through an entire codebase and translate it from Python to C++ or Rust — that requires hundreds of steps for a large codebase. How does it remember what it did, when it did it, what the current state is, what was changed? You can write that to a markdown file, but it also needs to understand the trajectory it went through. That trajectory is essentially the memory of an agent.

Ben Lorica: In your developer relations role, you talk to a lot of people at different companies. We’ve mentioned coding agents and deep research. What are some of the more common agents people are building, internally or externally?

Maarten Grootendorst: Aside from the obvious “it depends on the industry” answer — I do see coding agents being built quite a bit internally.

Ben Lorica: Okay.

Maarten Grootendorst: Yeah, just trying to see how they can prevent data from being leaked externally, because a lot of processes are very privacy-sensitive. I come from healthcare before I joined DeepMind, and especially in Europe—

Ben Lorica: I imagine if you’re in finance, at a hedge fund—

Maarten Grootendorst: Same, same. In those situations, people focus a lot on privacy and making sure everything stays within their own environments. You see a lot of people experimenting with local LLMs, using harnesses — could be Hermes, but also more foundational Python agent frameworks — and building around that. Or larger organizations that just use whatever cloud offering is available and build an agent on top. We’re so early in all of this.

Ben Lorica: For me, the areas where I see the most usage — and this won’t surprise our listeners — are still in the technical team bucket: DevOps, data engineering, platform engineering. They’re building agents to help them do their work. And in the background of a large website, there might be a bunch of agents doing heavy lifting, moving data around to get you the answer you want. But DevOps teams are building their own agents, data engineering teams are building agents for pipelines, and I’d imagine security teams are too, because they have to go through lots of log files.

Maarten Grootendorst: A question for you then — are they building full agents, or are they building skills? Because I’ve seen a lot of people focusing more on creating skills and giving those to whatever agent is available, rather than building agents from scratch.

Ben Lorica: I think internally, there are people building what we’d consider agents — systems that handle a huge chunk of their normal work, and they interact with it through prompting, but maybe don’t consider it fully autonomous. Just like people who use coding agents and actually know how to code might still test and read some of the code.

Maarten Grootendorst: Sometimes.

Ben Lorica: Yeah. And our listeners may be surprised, but there are large cohorts of people using coding agents who don’t know how to code at all — building websites and web applications. In the DevOps and data engineering fields, the agents being built are somewhat similar to coding agents: doing a lot of the work, but still with guardrails and humans in the loop. In non-technical fields like marketing or sales, what’s being built might be better described as skills — helping those teams be more independent from IT.

Maarten Grootendorst: Yeah, we can call them skills, workflows, pipelines, or just prompts.

Ben Lorica: Imagine you’re a marketing analyst at a Fortune 500 company, and your job used to be manually managing a bunch of ad campaigns and online campaigns. Now you can automate a lot of that work, have a dashboard to see what’s going on, and focus on higher-value things.

Maarten Grootendorst: Yeah, but I am curious about the long-term effects of all this — especially when you mention people coding without knowing how to code. That’s fun for a while, but in the long term, stuff breaks.

Ben Lorica: I’ve come across people who literally don’t know how to code, built a website, started getting customers, and then customers filed support tickets saying part of the site doesn’t work. Since they don’t know how to code, they go back to the same coding agent, the agent says it’s fixed, they tell the customer it’s fixed, the customer says it’s not fixed — and that’s when they realize they need to hire someone, because the coding agent can’t actually fix it. There are obvious dangers to going completely wild with these technologies.

So — open weights versus proprietary. This might be a sensitive topic since you work on Gemini, but you also work on Gemma.

Maarten Grootendorst: I work on Gemma. Ask me everything about Gemma.

Ben Lorica: In your day-to-day life — talking to friends, traveling, wearing your dev rel hat — what’s the level of interest in open weights?

Maarten Grootendorst: A lot. That’s mostly because I’m in Europe, and Europe loves to say, “We want to own things — we don’t want to hand everything over to someone else.” There’s way more interest in open weight models than I initially expected. There used to be quite a big performance gap, but now they’re closing in. These models are extremely capable. You can run them on MacBooks. When Llama came out, I saw so many threads of people buying Mac Studios just to run local LLMs. You see it across every part of the field — large organizations, small ones, finance, healthcare, all of it.

Ben Lorica: One challenge with open weights is that it’s a business decision, and business decisions can be reversed. Meta with Llama may not always produce open weights. Alibaba has been sending mixed signals. Some Chinese open weights providers are starting to do the same. It’s one thing to release an open weights model, but you have to release at a regular cadence, and that gets expensive. So where is the steady supply of open weights models going to come from moving forward?

Maarten Grootendorst: Yeah, I agree. Some organizations are pulling back from open source, doing less of it, focusing on different things — that’s understandable.

Ben Lorica: And one obvious advantage of open weights is that you can take the weights and run them in your own cluster. Enterprise teams get frustrated when they’re optimized for, say, Claude 4.5, and then it gets deprecated. With open weights, you have control. And I think teams are starting to realize they can use open weights for a lot of focused tasks — simple sentiment analysis, for example — where they don’t need the most expensive model, and they have control over it going forward. So while there’s concern about the supply of open weights models, for many use cases, maybe you don’t need the latest and greatest anyway.

Maarten Grootendorst: That can be the case. These models are very capable. I think there will always be a steady supply of open weight models. Looking at the current state of the field — Qwen is doing an amazing job, and we’re doing well with Gemma too.

Ben Lorica: Though Qwen lost a bunch of people, and there’s some worry Alibaba may pull back.

Maarten Grootendorst: I think they’ll continue. I don’t know for certain, but I think it’s still a very good strategy. And what we see with open weights is that you get so much back from giving models to the community. DeepMind is a good example, but the labs that have always given a lot to the community get a lot back in return. When people are excited about Gemma 4 — we released the 12B model two days ago — and you see them using it for cool use cases, driving research in directions you might not have anticipated, that’s incredibly valuable. You see things like DFlash, a diffusion-based drafter that’s super fast and being used with Gemma 4. Open weights allow someone without access to thousands of GPUs to still do very cool and interesting research. As long as I’m at DeepMind, I’m going to make sure we keep doing cool Gemma work.

Ben Lorica: All right, let’s close with a rapid-fire round. Keep each answer under a minute. Question one: what do you think about the trend around personal agents?

Maarten Grootendorst: I love personal agents — they’re very cool and interesting. At the same time, I’m very worried about security. We’ve seen people’s keys exposed, things deleted that shouldn’t be deleted. We’re in very early stages. Give it a little more time, and it will be amazing.

Ben Lorica: And run it locally with Gemma.

Maarten Grootendorst: Of course. I’m already selling too much.

Ben Lorica: Question two: reinforcement learning. I’m a big fan — I push out a post at least once a year saying it’s just around the corner. Now there seems to be a comeback with reinforcement fine-tuning. Are you paying attention to reinforcement learning?

Maarten Grootendorst: A lot. A few colleagues and I started something called “RL Back” with some bigger influencers like Jay Alammar and Josh Starmer from StatQuest, and we did a course on reinforcement learning quite recently. It’s such a cool technology — it’s the technique that makes LLMs what they are today. There are still a lot of new developments in that field: making them faster, more capable, multi-turn trajectories. It’s a hot area.

Ben Lorica: Third question: scaling laws. Anthropic in particular is big on scaling laws — more parameters, more data, that’s the road to better models. What’s your feeling about scaling laws right now?

Maarten Grootendorst: They change quickly. We started with “more parameters equals better model,” then shifted to reasoning — “longer reasoning equals better model” — and now we’re moving toward “longer trajectories equal better model.” More is better. They’re interesting, but they’re changing so fast that I’m wondering what the new scaling law will be in six months.

Ben Lorica: All right, closing topic: data centers. They’re a hot topic in the US — a lot of communities are coalescing around opposing data center buildouts. It’s complicated: if these AI technologies work and get adopted, we’ll need compute, or maybe only the wealthy will have access to AI. On the other hand, data centers require significant local resources — electricity, water, noise — and unlike factories, they don’t produce many jobs, especially with all the DevOps agents we talked about. What’s going on with data centers in Europe?

Maarten Grootendorst: We don’t like them. I’m speaking for the Dutch here — we generally don’t like them. That’s going to be very interesting going forward, because there’s still demand for AI. There are a lot of people who don’t like it, but there are also a lot of people using it, and we need to find a way to balance that. There’s no other path forward. I really hope we can focus more on efficiency when it comes to compute-heavy things. That’s why I focus so much on Gemma — small, capable models you can run on your phone, without needing large data centers, aside from training. We have to be honest: AI is here to stay. We just need to make it more efficient.

Ben Lorica: And with that — a closing note for our listeners: there are a lot of announcements about gigawatts of capacity being contracted, but if you follow what’s actually happening, not many data centers are being built and coming online. And with that, thank you, Maarten.

Maarten Grootendorst: Thank you. Bye.