What’s new in data engineering

The Data Exchange Podcast: Jesse Anderson, Ben Lorica, and Jenn Webb on new tools, processes, and best practices for building high-impact data applications.

SubscribeApple • Android • Spotify • Stitcher • Google • RSS.

This week our special correspondent and editor Jenn Webb organized a mini-panel composed of myself and Jesse Anderson, Managing Director at the Big Data Institute. Jesse is the author of a recent book entitled “Data Teams: A Unified Management Model for Successful Data-Focused Teams”. This conversation was focused on key areas in data engineering.

Let us know your favorite data engineering tools and your data engineering challenges. Take the 2021 Data Engineering Survey and get a free copy of the results and be entered into a drawing for a free Data Teams book & other prizes.

We discussed a range of topics including:

  • Streaming and real-time technologies and their applications to business intelligence and analytics.
  • Data management, particularly the rise of cloud data warehouses, lakehouses
  • The importance of “Ops” tools and processes, as exemplified by the growing interest in MLOps and DataOps.
  • The growing interest in workflow orchestration and workflow management tools like Apache Airflow and next-generation frameworks like Prefect, Dagster, Temporal, and more.

Jesse Anderson:

If we think about our Maslow’s hierarchy of needs of data engineering, we’ve now moved on from, “I just need to store some data”,  into  “just storing that data isn’t enough”,  to “now I need to make sure that I have a well orchestrated automation for that data to be cleaned and then have your ML pipeline  trained”. This is the level of maturity that we’ve gotten to,  there’s definitely this push within the companies to hit this level of maturity. …  Automation is a key part of what they want, they want reproducibility, that there is a button push, and it is done the same way each time, rather than here, we kick off this one-off thing that maybe somebody messes up on.


Related content:

2021 Data Engineering Survey

The 2021 Data Engineering Survey is now open and we need your help. The survey takes less than 5 minutes to fill out and we’ll share the report of the survey findings with you. You’ll also be entered in a drawing for free copies of the Data Teams book and other prizes.

[Image from pxfuel.]