The Data Exchange Podcast: Nikhil Muralidhar on lessons learned from developing and deploying machine learning models at scale.
Subscribe: Apple • Android • Spotify • Stitcher • Google • AntennaPod • RSS.
This week’s guest is Nikhil Muralidhar, a Graduate Research Assistant at Virginia Tech College of Engineering. He is the lead author of an excellent survey paper entitled “Using AntiPatterns to avoid MLOps Mistakes”. Nikhil and his co-authors provide a vocabulary of anti-patterns encountered in ML pipelines, with a focus on the financial services industry. In addition, they make several recommendations for documenting and managing MLOps at an enterprise scale. More specifically, they describe the crucial role played by model certification authorities within enterprises.
Nikhil Muralidhar:
We decided on three different subgroups of anti-patterns. The first set of anti-patterns has to do with model design and development. The second set of anti-patterns actually have to do with performance evaluation. Various problems or various inconsistencies that actually arise if the model performance is not evaluated correctly. And the third is, you know, what traditional MLOps people might actually call deployment and maintenance.… I would basically argue that this paper is sort of a celebration that machine learning has actually become so mature in the industry, that the reason we’re discussing all these things is because data driven discovery and data driven modeling has actually come to affect all of our lives in such a huge way.
Highlights in the video version:
- Introduction to Nikhil Muralidhar, a PhD Student about to graduate from Virginia Tech
What is the genesis around the paper “Using Anti Patterns to Avoid ML Ops Mistakes Paper?”
Nikhil Muralidhar’s background
What was the general methodology for assembling the list? How did you arrive at the list of anti patterns?
What is data leakage?
Time series and data leakage
Hyperparameter, Optimization, Turning
Machine Learning Models, Properties Significantly Affect Model Performance
Have your collaborators explored experimental management tools? What were your thinking as you were building the model?
One-Off ML, One-Off Projects, Second Forget Models
Documenting, Performance Evaluation, Best Practices
Anti Pattern: Perceived Empirical Superiority
No standard way to evaluate model performance and no standard data sets to evaluate
Computer Vision, NLP Communities, and Data Augmentation
Grading your own exam and perceived overestimation of performance
Alleviate Anti Pattern, Security Field
Penetration Testing, Machine Learning Models in Production
Responsible AI, Deployment and Maintenance of a Model
Data Crisis as a Service
Meta Data Management Services, Data Discovery
Recommendations in the Paper
ML Teams should focus on Data Quality, Pre Processing, and Feature Engineering to produce successful pipelines
Three components to ML Ops include: Monitoring, Automation, and Incident Response
Explain why teams should ensure humans in the loop of operational capability?
A model is meant to guide the human and ML is used to monitor
We’re far from having completely self learning autonomous systems
Develop ML Solutions to quantify decision uncertainty
What has been the reaction to the paper? Reaction from industry people and collaborators?
Reward System in ML Research and Domain Knowledge
Avoid User Bias
Nikhil, what’s next for you?
Link to Paper is in the Episode Notes
Related content:
- A video version of this conversation is available on our YouTube channel.
- “Model Monitoring Enables Robust Machine Learning Applications”
- “Managing machine learning in the enterprise: Lessons from banking and health care”
- Hamel Husain: “Deploying Machine Learning Models Safely and Systematically”
- Travis Addair: “The Future of Machine Learning Lies in Better Abstractions”
- Ryan Wisnesky: “The Mathematics of Data Integration and Data Quality”
- Rumman Chowdury: “The State of Responsible AI”
Subscribe to our Newsletter:
We also publish a popular newsletter where we share highlights from recent episodes, trends in AI / machine learning / data, and a collection of recommendations.
[Image: Sid Mosdell, Nature Pattern; usage via CC BY 2.0.]