Jeremiah Lowin on designing tools to allow teams to build, run, and monitor data pipelines at scale.
Jeremiah Lowin is co-founder and CEO of Prefect, the company behind the popular open source data and workflow orchestration system of the same name. We discussed the major design changes in Prefect 2.0, their move towards treating “code as workflows”, data engineering challenges facing data and ML teams today, and implications of looming trends in machine learning and AI.
Here are a couple of key posts on data engineering and orchestration written by Jeremiah:
Highlights in the video version:
- Introduction to Jeremiah Lowin
Root cause analysis, fault tolerance, and retry mechanism
Early orchestration tools and Python
Detecting when something is failing
Code as workflows
Building a company is about providing value
Software engineering rigor and no code/low code for data integration
Data scientists and notebooks
How does an orchestrator play in the analysts world?
How is the orchestration space coping with unstructured data?
- A video version of this conversation is available on our YouTube channel.
- Chris White: Towards a next-generation dataflow orchestration and automation system
- The Data Integration Market
- Distributed Computing for AI: A Status Report
- Pardhu Gunnam and Mars Lan: Why You Need a Modern Metadata Platform
- Michel Tricot: Modernizing Data Integration
[Image: Geometric Glass City Architecture Photo from Piqsels.]