Why Observability May Be AI’s Next Frontier

Ben Lorica

Ameet Talwalkar on Time Series Foundation Models, World Models, and the Shrinking Open-Weight Gap.


Subscribe: Apple • Spotify • Overcast • Pocket Casts • YouTube •  AntennaPod • Podcast Addict • Amazon •  RSS.

Ameet Talwalkar, Carnegie Mellon ML professor and Chief Scientist at Datadog, joins Ben Lorica to trace time series foundation models from skepticism to Datadog’s Toto V1 and V2. They cover why scale and data curation beat public benchmarks, the leap toward “world models” for observability, the shrinking gap between open-weight and proprietary models, and what AI means for the future of CS education and research.

Subscribe to the Gradient Flow Newsletter

Interview highlights – key sections from the video version:

Jump to transcript

Related content:

Support our work by subscribing to our newsletter📩

Transcript

Below is a polished and edited transcript.

Ben Lorica. All right, today we have an old friend, Ameet Talwalkar, professor in the Machine Learning Department at Carnegie Mellon, and also chief scientist at Datadog, which you can find at datadoghq.com. The taglines are: “Full-stack observability and security built for enterprise scale” and “See inside any stack, any app, at any scale, anywhere.” With that, Ameet, welcome to the podcast.

Ameet Talwalkar. Thanks, Ben. Always great to chat with you.

Ben Lorica. All right, so the first topic is time series foundation models, which will segue to what you folks have released and announced at Datadog. But just to set the baseline for our audience: foundation models, for most people, means I have a base model that I can then customize, post-train, and use for a bunch of things. The idea here is that people have built such a thing for time series. I may be misremembering this, but I think you helped run a workshop recently on time series.

Ameet Talwalkar. Yeah. My exposure to time series foundation models goes back a few years. It actually starts with my academic hat on, working with folks at CMU. There was a trend a couple years ago of folks in academia and elsewhere developing foundation models for X: time series, geospatial modeling, genomics, anything. And it kind of made sense. We have foundation models for text. Can we take that same idea of pre-training on massive amounts of unlabeled data in some sort of self-supervised way and magically get a foundation model that you can use either zero-shot – so you prompt it with a new data point that you want to make some prediction on – or fine-tune it, or do some post-processing on it, like you talked about?

Ameet Talwalkar. When I started looking at this a couple years ago, I think at that point I had been doing a little consulting with Datadog, but I had not joined in the capacity that I am in now. We actually asked the question from a very different point of view: to what extent are specialized foundation models actually working? Had they had their ClinicalBERT moments?

Ameet Talwalkar. Rewind back to 2018, when LLMs were not that big. What really made things pop for text foundation models was the idea of pre-training on the internet. These were small models in retrospect – 100 million-parameter models or whatever. NLP, as a research field, has always been very good at creating benchmarks. What happened with BERT and the models that came right after it was that this pre-training paradigm absolutely crushed all of the leading supervised models, which were each aimed at one particular benchmark or another.

Ameet Talwalkar. All of a sudden came BERT, where you had one model that you pre-trained, and then maybe you did fine-tuning for any of the benchmarking datasets, but it absolutely blew away all these other benchmarks. At the time it came out, it seemed crazy. How are you generalizing across these? We did not understand next-token prediction – not to say that we really do now, in terms of why it works – but empirically it was such a strong result. The baselines we were comparing against were all the leading models that everybody in NLP was looking at, so the leading experts in NLP could not ignore the fact that this new paradigm was way better than what they had been doing. As a result, they were early adopters.

Ameet Talwalkar. Fast-forward to when we started looking at this in 2023 or 2024. We asked the same question for time series, genomics, and a few other types of specialized foundation models: had they really hit their BERT moment? One thing we found was that the baselines people were using were not all that great. If I have a new approach, no matter what it is, and I compare it against something not very strong, I am going to look really good. But so what? Who cares?

Ameet Talwalkar. We had a paper that came out at ICLR last year. It seems like a day ago and also like forever ago. At that point, not being experts in any of these modalities, we came up with strong baselines for time series, genomics, and I think geospatial modeling. We took the latest and greatest, the leading specialized foundation models in those areas, and the benchmarks that people in those communities had been looking at, and we tried new baselines. What we found was that for all three of these domains, the specialized foundation models were actually no better than the baselines. The baselines were classical statistical approaches or classical ML approaches – much smaller, much simpler. In some cases, they were on par with the foundation models. In some cases, they were significantly better. So that was the negative result.

Ben Lorica. When you say baseline for a particular domain, how do you know those were the best models in that domain?

Ameet Talwalkar. We do not. The point is that they just needed to be strong. You never know what the best baseline is. But take time series, which is one of the areas we focused on. There are all these classical ARIMA models, moving average models, and so on.

Ben Lorica. Did the time series community have the equivalent of a leaderboard or something?

Ameet Talwalkar. I think there were two issues. One, it was a new community. I do not think anybody was trying to cheat. It was just a very immature community at the time. But at the time we wrote our paper, one big issue was the benchmarks. I think in time series a big issue was exactly that. The analogy I give is computer vision: think about MNIST, the benchmark Yann LeCun came out with in the ’80s or ’90s. Imagine trying to evaluate leading CNNs on MNIST in 2017. They would all be perfect, and maybe you could squint and see little differences, but they would not be meaningful.

Ameet Talwalkar. I think that was the issue with time series. The benchmarks were not strong enough to differentiate between good and bad. Nonetheless, everybody coming out with new models and saying, “We have a great new time series foundation model,” effectively had no evidence to demonstrate that they did, because the benchmark results were not actually interesting.

Ben Lorica. What year was that?

Ameet Talwalkar. I think that was the state of the world by the end of 2024. In 2025, we released Toto, but I would also say that…

Ben Lorica. When was that workshop that you ran? I remember that was in…

Ameet Talwalkar. December. That was in December of last year.

Ben Lorica. I looked at the program, and there were a lot of industry people in that.

Ameet Talwalkar. Yeah. This field has gone from very immature to rapidly improving. By the end of 2025, there were several models. We were one of them with Toto, but Amazon had Chronos, Google has TimesFM, Salesforce has the Moirai models, and there are a bunch of academic models. I certainly do not want to claim it was just us. But by the end of 2025, better benchmarks had come out. GIFT-Eval, a benchmarking suite by folks at Salesforce and others, came out. These were much more rigorous, more difficult benchmarks. The models also got better in various ways: they used more data and maybe more sophisticated modeling. We started to see in 2025 that compared to the same strong baselines, or any baseline you can think of, these foundation models trained on various types of time series data could get zero-shot results – meaning you do not do any post-training or fine-tuning – with better predictive accuracy than baselines that were typically supervised.

Ben Lorica. So the claim at that point, Ameet, is that they could zero-shot and outperform a skilled analyst who would sit down for a couple of hours to try to…

Ameet Talwalkar. That is a good question. I honestly do not know. For us, the way we approach this is that it is fun to ground the work you are doing in an emerging subarea of research, and we wanted to interact with the community. We still want to interact with the community. But a main reason we are doing all this is that there are very clear applications of time series foundation models in monitoring. We are going to talk about world modeling later as well. For our use cases, that setup just does not work.

Ben Lorica. The sheer volume. The sheer volume of time series that you have to deal with.

Ameet Talwalkar. No one can look at them. I actually do not know the answer to that.

Ben Lorica. I guess the question is, if I take one of these time series foundation models, zero-shot it, and then apply it to the stock market.

Ameet Talwalkar. Yeah. When we released Toto, we open-weighted it under a permissive license. Our idea was that we use the model internally for some product work, but we are not going to get into financial markets anytime soon. If other people want to use it, they can. It got downloaded a ton. Financial services companies are quite secretive, so I do not know this to be the case, but my strong guess is that they are looking for any edge they can get, and I am very confident they at least tried. Our model was one of the most downloaded models.

Ben Lorica. When you talk to people in this field now, the typical situation is that they work in companies like you, where there are millions of time series. In order to do any kind of analysis, forecasting, or anomaly detection against these time series, they would use automated tools based on classical techniques, and now these foundation models…

Ameet Talwalkar. That is right. A lot of our practically motivated work has been based on what we see at Datadog, and what you described is exactly that. The baselines we use at Datadog, or the things historically in production at Datadog, are exactly what you said. There is a question of whether we could replace them with zero-shot models.

Ben Lorica. When you ran this workshop, you interacted with people at other companies.

Ameet Talwalkar. Yes. I think you are generally right, but the field is new enough that a lot of people are still figuring it out. This is maybe a point I will make later in terms of what is next. The field of time series foundation models is intellectually interesting, and the benchmarks are better than they were before. But there is still this question of what time series is. It is hard to define. It is not one modality. Are financial data, observability data, weather data, and other data all one modality? Why is speech data or text not considered time series? They are not modeled that way; they are modeled more independently.

Ameet Talwalkar. So I think the field is pretty immature in terms of where these things will apply, where they should be used, and what the right baselines are for different things. You are right: for certain cases, the right baselines may not be ML or statistical models; they may be humans. But I also think you are right that in many cases, classical statistical models for time series forecasting are extensively used across domains. You could argue that anywhere those are used, these sorts of time series foundation models could potentially replace them.

Ben Lorica. Before the ImageNet moment in computer vision, there were people in computer vision working on things like SIFT and HOG, handcrafting features. What is the equivalent reaction from the existing time series community? How are they reacting to time series foundation models?

Ameet Talwalkar. I think with every passing month or year, people are embracing it more. I was pretty skeptical of it when I started to look at it. Again, I wrote an entire paper. It is funny: the workshop we went to at NeurIPS last year was literally titled – I do not want to say stolen, because they asked us and borrowed it with permission – but they basically took the title of our paper: have we reached the BERT moment for time series? There was a lot of skepticism, as there should be.

Ameet Talwalkar. But I think it is similar to what happened with NLP. At this point, every new benchmark that comes out shows these models doing much better. They are not perfect. There are certain aspects of traditional ARIMA-style models where they have more predictable behavior. What we saw in our more recent Toto work was that if you train for a certain context length, or a certain length of time series, the models work really well up to that length, or maybe a little longer. But eventually the predictions degrade in a way that classical approaches do not necessarily degrade, in terms of having somewhat predictable behavior. We have some long-horizon experiments where things eventually go haywire. There is room for improvement.

Ameet Talwalkar. But the evidence we are seeing now, not just from Datadog but more broadly within the community, is becoming hard to argue with. One other thing that surprised me, and this is very Datadog-specific, is that the time series foundation models we are looking at are still small in the context of foundation models. Toto V1 was on the order of 100 million parameters. The new models we are looking at go up to 2.5 billion parameters, but those are still relatively small. Nonetheless, they are still orders of magnitude bigger than a classical ARIMA model, which might have 20 parameters or 100 parameters. So you would imagine that inference-time costs are night and day.

Ameet Talwalkar. What I did not realize is that these ARIMA models are very small, but also finicky enough in terms of the hyperparameters you want to choose. What we often do, instead of having one model that we train ahead of time and then serve, is actually do the entire hyperparameter tuning process on the fly. It is small enough that you can do that, but it means that if you take on-the-fly hyperparameter tuning plus inference as the overall inference cost, it is actually cheaper for us to use a 100 million-parameter zero-shot model. It is easier because you do not have to do hyperparameter tuning; it is easier for a user to use. But it is actually faster at inference time than tuning an ARIMA model on the fly. That surprised me, and I think it provides more evidence for these things potentially being really useful.

Ben Lorica. A couple of questions. One is, when you zero-shot… Now, listeners, we are going to focus on what Ameet specifically built, which was much more optimized for what is called observability data. Maybe you should define observability data.

Ameet Talwalkar. Observability generally – and a big thing Datadog does – is monitoring the health of production software systems. Observability data is the data we are collecting. In some sense, it is the exhaust that any…

Ben Lorica. It is timestamped.

Ameet Talwalkar. Yes, it has timestamps. You are recording various metrics and various aspects of what is happening on every machine and across the network throughout your entire distributed system. We have a lot of metrics: CPU usage, latency, memory, all of these things across a bunch of different machines. There is some connectivity across these things, because there are network connections between machines. You might touch a database and then touch a particular service. As the system handles a request, it goes through the system in a certain way, and therefore there are dependencies.

Ameet Talwalkar. The point is that we have a bunch of observability data. Not all of it is classical numerical time series data, but that is a big part of it. That is called metrics, and that is what we are focusing on with Toto. When we later talk about multimodality and world modeling, the whole point is that our observability data is much richer than just time series data or just observed metrics, and we want to take advantage of that. But the starting point for the connection between us and time series foundation models is that we have a comical amount of time series data in the form of these metrics, which monitor the behavior of distributed software systems over time.

Ben Lorica. When you zero-shot it – before foundation models came about, when I first started playing around with these deep learning models for structured data and time series, I felt like they were so slow compared to the classical statistical models.

Ameet Talwalkar. Yeah, but the question is: when you were trying to use the classical statistical models, did you also account for the hyperparameter tuning costs in the slowness?

Ben Lorica. My assumption is that I already have the model, and then, boom.

Ameet Talwalkar. If you do that comparison, then it is night and day. A 100-parameter, effectively linear model is going to be way faster than a 100 million-parameter model.

Ben Lorica. So does that impact what you folks do?

Ameet Talwalkar. This is what I was mentioning. The way we typically deploy these ARIMA-style models, in order to get the bang for our buck, is by doing some amount of hyperparameter tuning. They are so cheap that you can do hyperparameter tuning, but they are also finicky enough that if you do not, the forecasts are not high enough quality. It turns out that it is actually cheaper for us to do zero-shot inference. Toto and these time series foundation models are very robust. You do not need to do much or any tuning. The whole point of being zero-shot is that it works out of the box. The cost of inference on a much larger time series foundation model is equivalent to, or oftentimes less than, the cost of tuning an ARIMA model per query.

Ben Lorica. What would be the intuition behind that? Is it because…

Ameet Talwalkar. If you do hyperparameter tuning, you are not just using a trained ARIMA model and predicting from it. You have some training set, and you are training a bunch of different models on the fly. Each of these models is very fast to train because they are small, but if you are doing this a bunch of times at request time, the total time is still not that much – it is certainly under a second, because they have to be fast – but it is not instantaneous. It is hundreds of times more work, at least, and probably orders of magnitude more work, to get the model you want than it is to serve it. Since you are doing both and lumping that together as the inference cost, serving a 100 million-parameter zero-shot model is perfectly fine for us.

Ben Lorica. You now have two generations of this foundation model. What is the main difference between the two generations?

Ameet Talwalkar. There are two main differences. The biggest one is scale. With the first version, we proved to ourselves that we were among the first to show – not the first to show, but among the first – that we had achieved this BERT moment. You can really crush supervised baselines with a zero-shot model. Very cool. But these models were really small. They were BERT-level, around 100 million parameters.

Ameet Talwalkar. I would argue that this new release, Toto V2, which was a couple of months ago, was sort of like the GPT-2 moment: the idea that you can actually scale things up. As much as possible, we are borrowing the amazing stuff that has happened with LLMs. A 100 million-parameter BERT model is tiny. GPT-2, in retrospect, is also tiny, but it was orders of magnitude larger. The bitter lesson is that more data and larger parameter counts lead to better results.

Ameet Talwalkar. The main goal with the next version was to check on scaling. We have a ton of data. We were barely using any of our data with Toto V1, and we are still using a very small subset of our data with Toto V2, but we had a lot more data we could use. The question was: as we throw more data into this, can we get better results? It is nontrivial to get this to work. It is more expensive to do the training and inference, and you also have to figure out a bunch of different hyperparameter tuning. The upshot is that we were able to very definitively show that bigger models are significantly better. We stopped not because we were saturating – the end models were 2.5 billion parameters, again quite small – but because we felt we had shown enough. They are not saturated. There is way more to do. I think you can push it to be much bigger. I do not know where it will end. That was the biggest takeaway.

Ben Lorica. A technical scaling law?

Ameet Talwalkar. I am careful about saying scaling laws. Scaling laws are more about generating a prescription: you train a bunch of different models at different trade-offs between model size and amount of data, and you want to know the right way to scale up. If I am trying to train a really big model, how should I trade off model size with the amount of data I train it on? We were not quite doing that. We were basically asking: do bigger models do better? Even with LLMs, scaling laws came after people had already established that bigger is better. The question is, what is the right way to scale up? So not quite scaling laws, but yes, scale helps.

Ameet Talwalkar. The other thing that was interesting goes to your point about observability data and metrics. Weirdly, even though our interest is observability data and metrics, many evaluations have shown that we are state of the art on general-purpose time series benchmarks as well as our own benchmark.

Ben Lorica. So this foundation model that was specifically trained for observability data is doing well on general-purpose time series benchmarks?

Ameet Talwalkar. Yes. It is state of the art. It is beating everybody by a lot. That was true with Toto V1 too.

Ben Lorica. Did you hear that, Google, Amazon, Salesforce?

Ameet Talwalkar. No, I mean, they are all very good. I am sure they will catch up. With Toto V1, we were beating everyone by a lot too, and then six months later we were no longer doing that because people caught up.

Ben Lorica. But you are focused on observability, so this is a side effect.

Ameet Talwalkar. This is a difference between the first and second versions. In the first version, we trained on a ton of observability data, but also a lot of general-purpose time series data. In V2, we literally only trained on…

Ben Lorica. General-purpose time series data?

Ameet Talwalkar. There are publicly available general time series datasets. I cannot remember what the collection is called, but it is similar to the data other folks in the community are using or collecting. The second big thing is early evidence that data mix matters a lot. Again, this is an analogy with LLMs. Data curation is going to matter. We are just at the beginning of this. What we did was a bit ad hoc: we explored what mixes might work better or worse. Surprisingly, we found that excluding the public data entirely led to better results for us.

Ben Lorica. Excluding the public data led to better results?

Ameet Talwalkar. Yeah. You can have a held-out dataset, whether it is an observability benchmark or a general-purpose benchmark, and we found that the public data, at least the way we were using it, was not helping us.

Ben Lorica. But your focus is on observability, right?

Ameet Talwalkar. No, I am saying even in the context of public benchmarks, the public data was not helping us. It could be that we were not doing data curation the right way. The order in which you train on data may matter.

Ben Lorica. Maybe the public data is the equivalent of Reddit garbage.

Ameet Talwalkar. Who knows? But I do think data curation is going to be a big thing in the future for exactly these reasons. We are a little more advanced now. We have hit the BERT moment. We have hit the GPT-2 moment. There are a lot of future milestones: scaling laws, better data curation, scaling significantly further. There is a lot more to do.

Ben Lorica. Obviously, the second generation of the foundation model just came out, but the first one has been out for a little while. Have you heard stories of people post-training it for their own purposes?

Ameet Talwalkar. The biggest thing is, I do not know how meaningful it is, but it has been downloaded on Hugging Face a ton – over 10 million times. At one point, it was one of the most downloaded models.

Ben Lorica. Have you heard stories?

Ameet Talwalkar. There have been some. None immediately come to mind. I should have prepared for that.

Ben Lorica. If you were not at Datadog and you took Toto, would your intuition say that I can take that, post-train it on my own metrics, and it will be better?

Ameet Talwalkar. Internally, we have found that ourselves. While zero-shot performance is still quite good, for particular problems where we want to use it, even if it is subsequently in a zero-shot setting, fine-tuning in various ways on more in-distribution data for a particular application has really helped us. There was one example around resource allocation or resource forecasting. I cannot remember the exact example.

Ben Lorica. This was not in the training set?

Ameet Talwalkar. The particular thing I am talking about involved seasonal effects. Suppose you are buying GPUs and want to forecast your GPU needs in the future. Let’s say your costs spike on the first day of the month because that is when you renew your monthly contract. That sort of data was not in our training set, at least not enough, and the model did horribly because it completely missed these spikes. Fine-tuning really helped and allowed us to do better for that sort of application.

Ameet Talwalkar. I think the analogy with normal LLMs is going to hold. These things are already shockingly good, even though they are early, at general-purpose zero-shot forecasting. But if you have a specific problem you care about, you will probably do better by fine-tuning on data that is very in-distribution. The models are still small enough that fine-tuning 100 million- or even 2 billion-parameter models is not hard.

Ben Lorica. Carrying this analogy with traditional LLMs further, one of the first things people tried with traditional LLMs was providing additional context, whether through RAG or whatever. Nowadays with agents, there is tool use. Would any of those make sense in the context of this type of foundation model? In the case of tool use, I guess you would call a specialized forecaster.

Ameet Talwalkar. We have not looked at tool use yet, though I think that is reasonable. Two things come to mind. One is the notion of auxiliary variables. Seasonality is a good example. Are there ways to incorporate seasonality data if you know the thing you care about is very dependent on seasonality?

Ben Lorica. You mean in a RAG fashion?

Ameet Talwalkar. There are different ways to do it. I think the folks at Amazon, with Chronos or Chronos 2, trained on a bunch of data that included various auxiliary data, such that their model was able to somehow automatically handle some of this auxiliary data. What we provided was more along the lines of RAG, if you stretch the analogy. We did not do anything at training time for this, but we provide functionality that allows you at test time to feed in your auxiliary variables. They are used to influence the forecasts, and that can really help. We have a blog post about that.

Ameet Talwalkar. The other thing is multimodality. That is where I keep going back, because for us…

Ben Lorica. Let’s pivot to the notion of world models, after you define it.

Ameet Talwalkar. World models are the extreme of multimodality. The first thing we do with multimodality is combine text and time series. Maybe you want to ask questions of your text. Maybe you also have metadata in the form of notes describing the time series. You want to be able to query the time series and ask questions about it: is there an anomaly here? What is going to happen next? You also want the model to potentially learn from any textual context associated with the time series.

Ameet Talwalkar. We created an open source benchmark anchored in observability problems. In incident reporting, when your production software system goes down, that is an incident. You want to figure out what happened. We have a bunch of data. Our benchmark consists of data derived from real incidents, where a very natural thing to do is ask questions of your time series. We created that benchmark, open sourced it, and evaluated it with various frontier models. These models are surprisingly multimodal, and there are various ways you can get them to reason about time series. We also did very simple post-training of an LLM glued together with Toto – via fine-tuning or RL – and found that the specialized model did significantly better.

Ameet Talwalkar. In both cases, the idea is that time series on its own is great, but it is part of a broader story. Thinking about how to integrate different modalities is important. Time series plus text is the most common thing in this field right now that people are trying to integrate across various domains. Healthcare people ask about that a lot too. You might have a patient chart with notes on it. How do you ask questions about a new test result? There is growing work there, and I think that is really important. Where we are going is more along the lines of world modeling for observability.

Ben Lorica. In that case, you have a bunch of data sources that are very specific to your domain: source code, logs, traces, topology, network topology, events, alerts.

Ameet Talwalkar. Exactly. We have all this data anyway. Metrics are the classical time series, but we also have logs, topology, code, events, alerts, traces – whatever it is.

Ben Lorica. The goal here is to build the model that you can use for alerts, simulations, and downstream applications.

Ameet Talwalkar. There are a bunch of downstream applications, but the ambitious goal is to view this as a world model. By world model, I mean we want to simulate the behavior of a distributed software system in the same way that a world model for the 3D world is trying to predict future events in a world. We want to be able to say there is a current distributed system where we are getting a bunch of observations in the form of different telemetry data – metrics, traces, and other modalities. We want to model some underlying state of the system so that we can predict or generate what that same telemetry data might look like in the future, potentially given an event that happened to the system.

Ben Lorica. I have talked to a bunch of world model people, and like you said, the goal is to predict future states. But it does not seem like they are at the point where they can predict that far in the future.

Ameet Talwalkar. World modeling is very hot these days, and there are a lot of different discussions about what…

Ben Lorica. Mostly in robotics.

Ameet Talwalkar. Mostly in robotics. I think progress is being made, but it is certainly not a solved problem. In observability, as far as I know, we are maybe the only people working on this now, or at least at a certain scale, with a certain amount of data and resources. We are certainly very early. I do not want to pretend we have solved this problem. We have not.

Ameet Talwalkar. What we have done is identify what modalities we care about and develop some idea of how to model each of these modalities separately. At this point, we are experts at time series. For text, there are open-weight models. For some of the other modalities – topologies, traces, and logs – we are also doing separate modeling work. The idea is how to put these things together in a unified way so that we can make progress toward this ambitious goal.

Ameet Talwalkar. Another big part of this is benchmarks.

Ben Lorica. Benchmark BOOM, indeed.

Ameet Talwalkar. Yes. BOOM is a great starting point, but here we want benchmarks that capture different things. Some of the benchmarks should capture the next state of the system and figure out how to measure it. We can leverage a lot of the data we have at Datadog to do this. One way to think about it is that there are points in the system where we know an event has occurred. Maybe you updated or changed the system, or something broke. You want the system to be able to predict, given the before state, what is going to happen after, given knowledge that an event occurred, and create benchmarks that measure that.

Ameet Talwalkar. We also want to use the same model to perform a bunch of different downstream tasks, such as various forms of detection. We want to be able to detect that something is broken in the system as early as possible. There are two variants. One is proactive detection, which means the problem already happened, but it is better to go to the emergency room a minute after a heart attack than 10 hours later.

Ben Lorica. For our listeners, in Ameet’s world of incident response, the goal is to lower the mean time to recovery.

Ameet Talwalkar. Currently, the goal is exactly to minimize downtime. The even more ambitious goal is to avoid downtime altogether.

Ben Lorica. Right.

Ameet Talwalkar. Again, I do not think we are there yet, but that is preemptive detection. Can I model a system so well that…

Ben Lorica. In classical time series, the band gets wider.

Ameet Talwalkar. Right. The longer into the future you go, the harder it is to predict. But even if we have a five-minute, or very ambitiously an hour, head start, that is huge. That could mean that as soon as the problem happens, we have identified it. Or maybe five or 10 minutes before it happens, we can say with high probability, “Something bad is about to happen.”

Ameet Talwalkar. A world model should also help with what-if analyses. What happens if I scale down this service? What happens if I shift traffic from one region to another? How might something break, or how might things change? The degree to which you can model that is really cool. A third part is agentic training. A big part of RL post-training at Datadog, or anywhere else, is creating realistic simulations that you can use to train your model. You would imagine that this sort of world model could generate really realistic simulations, the better it gets at doing world modeling.

Ben Lorica. Generally, in both world models – which are obviously just beginning – and time series foundation models, you have interacted with a bunch of people through this workshop and other forums. Is your impression that people are basically all doing the same thing? Are the architectures the same? Does no one have any secret sauce?

Ameet Talwalkar. In traditional time series forecasting, yes, people are converging. In the context of Toto V2, a big thing we did very transparently was say: there are really smart people at other places releasing these models, so as much as possible, let’s borrow the best ideas. It does not matter where they come from. People are starting to converge.

Ameet Talwalkar. In world modeling, even forgetting about observability, it is early, so that level of convergence has not happened. Yann LeCun has a very JEPA-style thing. People have different opinions about what will work and what will not. Nothing has completely worked yet such that everybody has converged on it. But I can imagine that will happen over time as we are successful.

Ben Lorica. All right, two quick topics to close. The first is open-weight models. As you know, I think the consensus now is that open-weight models, mainly from China, are maybe on the order of three months behind, maybe 12 months at the outer edge. As far as open-weight models in general, most of the best ones are really coming from China. Google has a family called Gemma, which is mainly aimed at edge devices.

Ameet Talwalkar. They are great, but they are really small.

Ben Lorica. Is my assessment fair? The gap between the open models, open weights, and the proprietary models is now down to months?

Ameet Talwalkar. It is hard to say, but I would agree with the three- to 12-month range.

Ben Lorica. And to be fair, whatever they are doing, whether benchmarking or distillation, it is working.

Ameet Talwalkar. Yes, but I would separate those two things. Benchmarking is not great if they are just benchmarking, because then they have a nice tweet and say, “Look how close we are.”

Ben Lorica. But that is my point. There is a lot of discussion about, “Oh, they are just benchmarking,” and I am sure they probably are to some extent, but the models themselves are actually good enough that people continue to use them.

Ameet Talwalkar. Exactly. The distillation thing is more…

Ben Lorica. And to be honest, wearing your CMU hat, at least they have open weights that you as researchers can work with.

Ameet Talwalkar. It is great for researchers. It is great for enterprise. GLM-5.2 just came out, and people are very excited about it. It is very good. The benchmarks are great, but people are actually starting to use it. We are also seeing internally that it is a high-quality model, which is nice. We have more control over it than calling a third-party API that we do not know anything about.

Ben Lorica. Here is another claim I am going to test with you. On the coding front, the combination of the harness, the data, and the loop that the proprietary labs, OpenAI and Anthropic, have gives them maybe a bigger edge on coding than elsewhere. Is that a fair assessment?

Ameet Talwalkar. I would say yes, but it also seems like there have been incremental improvements. Whatever happened with Opus 4.5 – whichever one was released around Thanksgiving of last year – fundamentally changed things. All of a sudden, coding agents, not that they did not work at all before, started working well enough that people said, “Okay, everyone is going to start using this.” But that is still in line with this three- to 12-month window. It is hard to predict.

Ben Lorica. Even in coding, you think it is only three to 12 months?

Ameet Talwalkar. I do not know. The three to 12 months has not happened yet, so we will see what the coding models look like in a couple of months from the open-weight side. Maybe they will be good enough. But the state of the world today is certainly that your assessment is accurate.

Ben Lorica. How is this playing out for enterprises? Enterprises are realizing three things about proprietary models. One, they are expensive. Two, you are potentially sending IP to an endpoint. Three, you do not actually have control over your roadmap because they change things, sunset a model, or you lose access to a model. My prediction is that more people are going to start taking some of these open-weight models and controlling their own destiny.

Ameet Talwalkar. I hesitate to make predictions, because we have been in this world where we keep oscillating back and forth between saying the frontier labs are so far ahead that everyone else should just stop, and saying the frontier labs are dead.

Ben Lorica. I would say for 80% of the tasks you need, you can get by with the open weights. For the remaining 20%, sure, send it to the…

Ameet Talwalkar. I do think that the moat for a lot of enterprise companies is their data, and the way they can use their data is through some sort of post-training. You are definitely seeing this trend. This is what we are doing in our lab at Datadog. Harvey is doing this. Cursor did this. Fin from Intercom did this.

Ben Lorica. I just did a deep dive on reinforcement learning startups. There are close to 30 of them doing exactly reinforcement fine-tuning to help companies.

Ameet Talwalkar. Exactly. This is becoming a thing. If you have a lot of data and think that is a moat and an edge for you, can you use the latest and greatest open-weight model to…

Ben Lorica. And the cost, once you start hitting these models hard, is big.

Ameet Talwalkar. That is certainly the vibe right now. Things keep changing so quickly that I have decided making strong predictions about what things will look like in a few months is how you end up looking silly. Every six months, the world looks completely different. It is very exciting.

Ben Lorica. The thing that worries me about open weights is that open weights is a business decision, and business decisions can change. Look at Llama – no longer open. The Chinese models right now are open. I do not know for how long.

Ameet Talwalkar. To me, and I think one thing we wanted to talk about at the end relates to academia. I do wonder if this would require a big effort – bigger than any one lab or even one university. In physics, and I am not a physicist, there are particle accelerators. These are massive-scale projects that require a lot of government funding, billions of dollars. But they allow people to work on things no one person can work on.

Ameet Talwalkar. There is an argument that a truly open source endeavor – not just open weights – crowdsourced by everybody and maybe funded by the government, is the way to protect against exactly what we said. This would not be a business decision. This would be: we want, from a scientific perspective, to truly understand what these things are doing, to be able to poke at them and get access to everything, and not just be dependent on benevolence from companies that are trying to compete with each other, not just do research. I think that is potentially a really attractive platform. It would take a lot of organization and effort, but it is a lot safer than hoping it will remain in the interest of a company to continue opening models.

Ben Lorica. There will be more to announce on that front from a mutual friend of ours, which I cannot disclose on the podcast, shortly. Closing topic: the future of computer science. Obviously, headlines suggest that the actual task and occupation of coding might be in trouble. Enrollments are declining across the board. What say you, Ameet? Before you answer that, tell us about enrollment. Is it really declining?

Ameet Talwalkar. Honestly, because I am on leave right now, I just do not know. I have heard the same things you have said, but I do not know what CMU’s numbers are. There is definitely, understandably, concern about CS and AI more broadly.

Ameet Talwalkar. We met in the early 2010s, and I remember being at Berkeley. Mike Franklin was the head of the AMP Lab at that time, and maybe he was department chair – I might be off by a year or so. There was this big tension where CS was driving up all of the enrollment at Berkeley. A lot of the revenue was coming in through CS, and within CS, a big portion was people excited about machine learning. In the last five years, pretty much all of the excitement in CS has been through AI, at least in terms of people moving with their feet and wanting to move into this area.

Ben Lorica. Let’s give a shout-out to our friends in systems as well.

Ameet Talwalkar. Yes. Software engineering was viewed as the most stable job and so on. My broad take is that in the context of AI, what has happened in the last 10 years has led to a really great but unstable dynamic for CS academia, whether it is CS versus the rest of academia or CS academia versus industrial CS. With such growth, and with AI in particular, we were in a world where the leading companies – Google, Microsoft, Facebook – had increasingly strong research labs that were very open and interactive with academia.

Ameet Talwalkar. We had this nice ecosystem. They might have had more resources, but we had a lot of freedom, we had and still have brilliant students, and we were all working together. It seemed like you could have your cake and eat it too. You could do really cool research, have immediate translational impact, and start a company based on your research right away.

Ben Lorica. You could spend a day at Facebook one day a week.

Ameet Talwalkar. Exactly. Everybody was doing that. It was awesome. I started a company in 2017. After we sold it, I worked at a VC firm part-time, in part to help strengthen their academic network, because the understanding was that a nontrivial amount of the innovation in AI that would be immediately commercializable would come from academia.

Ameet Talwalkar. The obvious thing that has changed is that AI researchers, more broadly in industry and academia, succeeded and came up with something incredibly commercially viable. We are so early, but LLMs have moved beyond research. They are the thing everyone is talking about everywhere in society, let alone in academia or industrial research. They are being studied and developed at a scale that academics cannot, or should not, try to match.

Ameet Talwalkar. In some sense, we need to decouple a little bit. When you are growing, everything is exciting and new problems emerge, but it is a good problem: we are growing. When you are no longer growing, or potentially need to reduce because some of what you have done has graduated to industry, it does not feel great. But that does not mean academia is going extinct tomorrow.

Ben Lorica. Most of the practical work can only be done in industry because of compute and data.

Ameet Talwalkar. Exactly. My take is that AI research in academia, and maybe CS in academia, will need to rethink a lot of what we are doing. It will still have a lot of value. The teaching part may be more important than ever, because AI is changing every day. We need to educate more people to use this.

Ben Lorica. Are you still going to need to teach people how to code?

Ameet Talwalkar. Yes, but maybe in a different way. The other thing you are hearing is that software companies are not hiring as many junior people because junior people do not have the experience in how to shepherd coding agents. They do not have the experience of coding themselves that allows them to work well with and guide coding agents. But who is to say that future education cannot teach those skills? I do not have the answers. There is an undercurrent here: LLMs have changed things a lot. Math is another example. Leading LLMs are so good at math – maybe not at every kind of math problem, but they can execute on arbitrarily hard math problems that are well posed.

Ben Lorica. The problem-set, problem-solving kind of math. Can they be beyond that?

Ameet Talwalkar. Even frontier-level research, if it is well posed, they can.

Ben Lorica. You still need to check it, though.

Ameet Talwalkar. You do need to check it. But I was talking to a colleague of mine, a very well-established theoretical statistician, who was saying that even a few months ago, he used to check it and get excited, but it was always wrong. Now he says it just does not make mistakes in the same way. It is crazy. He can take an idea, prompt it with the idea and a half-baked direction, and solve problems he wants to solve way faster. It is not doing all the work for him, but it is accelerating things in an amazingly exciting way.

Ameet Talwalkar. If this is where it all ended and we froze things here, I think we would figure everything out. Another undercurrent, though, is that AI keeps getting better, and people are understandably worried about what happens longer term when AGI comes.

Ben Lorica. Does this change the definition of who is good? My concrete example is this: imagine that in 2036, the person who wins the Fields Medal is not someone like Terence Tao, who has the raw IQ, but someone who knows how to use these tools well.

Ameet Talwalkar. I think 100% yes. That in itself is disorienting and hard for people who are currently here. But rules change in anything, or the situation changes, and communities in general adapt. For every individual, it is hard because some people are suddenly at a great advantage or disadvantage because different skills are prized.

Ben Lorica. These models in particular are great at synthesizing and making cross-disciplinary observations. One of the problems with traditional academia is that if you are in math, you may only do number theory. You may not read papers in other fields, so you do not connect the dots that these models are able to connect.

Ameet Talwalkar. Right.

Ben Lorica. Maybe in some of these other fields, and actually in computer science, people will end up building much better systems because they now have access to tools that can connect the dots.

Ameet Talwalkar. My friend is a theoretical statistician, not quite computer science but adjacent. He is currently very happy that he is much more productive in what he is able to do. There is an education question: he is a tenured professor at CMU who has learned to do all of this work himself, so he is very experienced and knows how to guide these things. How do you teach a first-year PhD student in statistics to get to that point? That is a real question. But again, I think it is solvable once we acknowledge that there is a problem and that things need to be different. This is why teaching is super important. I do not know how to do this, but I am confident that collectively, as academics and teachers, we can and will figure it out.

Ben Lorica. A lot of things changed. For example, the things you might in the past have given to a grad student, maybe you just give to an LLM.

Ameet Talwalkar. That is absolutely true, and that is the source of a lot of the concern. But I would also argue that everybody, whether you are a new student, a tenured professor, or whoever you are, should lean into using these things. That is the cliched thing everyone says, but it is true. Things are changing so fast. If things were frozen right now and we knew the lay of the land, we would adapt. The problem is that the lay of the land keeps changing every six months. We do not know what the endpoint will be. The best thing people can do is lean into using these things so they actually know what they can and cannot do with them.

Ben Lorica. I know what the endpoint is: recursive superintelligence, recursive self-improvement. It is the new buzz.

Ameet Talwalkar. That is what many people say. We will see what that is and what it looks like.

Ben Lorica. We are looking forward, Ameet, to the day when Toto is going to be RSI.

Ameet Talwalkar. We want self-healing distributed software systems. That is what we want automated.

Ben Lorica. We want Toto 3.0 to build Toto 4.0.

Ameet Talwalkar. Yeah. We will see.

Ben Lorica. Thank you.

Ameet Talwalkar. Thank you. Thanks a lot. Bye.