Amit Sharma and Emre Kiciman on making causal inference and learning more accessible.
Subscribe: Apple • Android • Spotify • Stitcher • Google • AntennaPod • RSS.
This week’s guests are Amit Sharma (Principal Researcher) and Emre Kiciman (Senior Principal Researcher) of Microsoft Research. We talk about practical applications of causal inference, a set of tools and techniques that enable data teams to draw causal conclusions based on data. Amit and Emre are part of the team behind DoWhy, a new open source library for estimating causal effects based on historical data alone, particularly useful when we cannot run an experiment because of time, expense, or ethical concerns. In this episode we discuss the growing interest in causal inference, and we do a deep dive into the state of tools (also see this paper on DoWhy).
Amit Sharma:
So whenever you have a decision, for example, whether the vaccine works or not, or what kind of economic policy a country should adopt, these have been historical decisions, where causality has traditionally been applied. But today, what we are seeing is with AI systems being deployed in many different settings, the same kinds of questions are also coming up for those systems. Let’s say when you are recommending certain jobs to people, or when you’re recommending financial decisions, we want to make sure that those decisions also have the right impact for users. Then there are also questions about fairness and robustness of systems. Causal inference is the sibling of prediction in terms of scientific inquiry. … Causality comes in when you want to ask, “I see the system proceeding in a way I now want to change the systems, I want to make the system better, what should I do, so that my intended action actually also has the right consequence on the system.”
Emre Kiciman:
What we saw is other libraries focused really on statistical estimation. In some sense, that makes sense. That’s the hard part to code. The other pieces are easy to code if you know what to do. But our end-to-end solution really is aimed at data scientists who want to use causal inference.
Highlights in the video version:
- Introduction to Amit Sharma and Emre Kiciman
What is causal inference?
What sets causal inference apart from traditional approaches to modeling?
Prototypical causal inference use cases
Introduction to DoWhy and major components of a practical causal inference library
Who is your intended user? For what uses? Who are you targeting?
The workflow includes four steps: specify assumptions, identification, statistical estimation, robustness and validation
As you look ahead on the roadmap, what part of this four step process will you focus on?
Key Metrics: community of contributors, pip downloads, users success or failure stories
What’s the state of causal inference in academia?
Causal Inferen in relation to Explainability and Fairness
Will causal reasoning be another added capability of data teams?
Adoption of causal inference
Related content:
- A video version of this conversation is available on our YouTube channel.
- Christopher Nguyen: “What is AI Engineering?”
- Che Sharma: “Modern Experimentation Platforms”
- Nic Hohn and Max Pumperla: “Reinforcement Learning in Real-World Applications”
- David Blei: “Topic Models: Past, Present, Future”
FREE report:
[Image: Newton’s Cradle from Wikimedia.]
.jpg){kind=link}