Modern Experimentation Platforms

The Data Exchange Podcast: Che Sharma on how seamless end-to-end experimentation workflows supercharge product development.

Subscribe: Apple • Android • Spotify • Stitcher • Google • AntennaPod • RSS.

Che Sharma is the founder and CEO of Eppo, an experimentation framework that integrates with modern data platforms (cloud lakehouses and cloud data warehouses). We discuss the importance of investing in experimentation tools and the power of having a well-oiled experimentation culture within an organization. Che also explains how modern data platforms enable a variety of applications, including experimentation frameworks like Eppo.

Modern data platforms have spawned a growing ecosystem of data engineering tools for areas such as data quality, data discovery and governance, data integration and much more. Meanwhile, low-code and no-code tools are making BI, analytics, and machine learning accessible to a broader range of users. Investors and engineers have created a glut of ML infrastructure and MLOps startups, but tools higher up the stack, like experimentation, have not attracted as much attention. It’s not out of ignorance: there are numerous books that extol the power of experimentation ([1], [2], [3]).

Up until now, experimentation platforms have mostly been bespoke solutions found primarily within technology companies. Modern data platforms make it feasible to build solutions that will enable companies to democratize and systematize experimentation. In light of the potential impact of an experimentation platform on a company’s culture and bottom line, I hope we see more accessible tools in the years to come.

Che Sharma:

The key thing with experiments is to know, when are you helping and when are you hurting. We’ve seen a number of papers that have basically acknowledged this one point, which is no matter where you are, once you start measuring things, you realize of the 100 experiments you were running that you thought were going to improve things, probably only 30% of them actually improved things, another 30% made things  worse. So once your eyes are peeled back, and you start to see these outcomes, it becomes crucially important to know which experiments are happening and to know it comprehensively across the org.

The reason I started an experimentation company is, on top of that kind of tangible ROI, there’s so many amazing second order effects that happen from establishing an experimentation program. So one of the ones that I felt over and over with every company is, before you run experiments, if you’re a data team, you primarily are living in the world of reporting. Maybe some ML models, but primarily reporting. The problem with reporting is it doesn’t tend to influence product decision making very much. Take an example where you have some top line revenue dashboard, and you are saying, here’s how much money we’re making every year. That’s great for the CFO for the board, they can look at these numbers. But if you’re a product manager, you stare at that dashboard, and you wonder what to do with that … But once you start running experiments, suddenly every product team has a really intimate connection with metrics. A major goal for every data team is to kind of build this muscle of kind of leveraging metrics in decision making.

… But the thing that gets me most excited about experimentation is that once you nail fluid, seamless end-to-end experimentation workflows, it creates this entrepreneurial culture, where suddenly people can take risks, you know, they can try things out and see if customers like it or not. It’s this amazing culture that once you’ve been at a place like an Airbnb, or a Netflix, you know, you really start to appreciate that you can have bottoms up product development.

Highlights in the video version:

Related content:

• A video version of this conversation is available on our YouTube channel.
• “Model Monitoring Enables Robust Machine Learning Applications”
• “Taking Low-Code and No-Code Development to the Next Level”
• “Data Quality Unpacked”
• Hamel Husain: “Deploying Machine Learning Models Safely and Systematically”
• Nikhil Muralidhar: “MLOps Anti-Patterns”
• Travis Addair: “The Future of Machine Learning Lies in Better Abstractions”

Subscribe to our Newsletter:
We also publish a popular newsletter where we share highlights from recent episodes, trends in AI / machine learning / data, and a collection of recommendations.

[Photo by National Cancer Institute on Unsplash.]