Dataflow Automation

Jeremiah Lowin on designing tools to allow teams to build, run, and monitor data pipelines at scale.

SubscribeApple • Android • Spotify • Stitcher • Google • AntennaPod • RSS.

Jeremiah Lowin is co-founder and CEO of Prefect, the company behind the popular open source data and workflow orchestration system of the same name. We discussed the major design changes in Prefect 2.0, their move towards treating “code as workflows”, data engineering challenges facing data and ML teams today, and implications of looming trends in machine learning and AI.

Here are a couple of key posts on data engineering and orchestration written by Jeremiah:

Highlights in the video version:

Related content:

FREE report:

[Image: Geometric Glass City Architecture Photo from Piqsels.]