Modernizing Data Integration

The Data Exchange Podcast: Michel Tricot on key components and features of modern data integration solutions.

SubscribeApple • Android • Spotify • Stitcher • Google • AntennaPod • RSS.

This week’s guest is Michel Tricot, co-founder and CEO of Airbyte, a startup behind the popular open source project with the same name. While still a relatively young project, Airbyte is a favorite among data and platform engineers who are responsible for building and maintaining data integration systems within companies.

Download the 2022 Data Engineering Survey Report and learn how companies are designing and building data platforms.

As we observed in recent posts on DataOps and Metadata Management Systems, several trends are nudging companies towards modernizing their data infrastructure: data volume and the number of data sources have exploded, the number of teams and personas that depend on data continue to grow, these new users of data expect a certain amount of reliability and freshness, and finally regulators and users are increasingly concerned with issues related to data privacy and security.

Michel Tricot:

I always go back to how we think about building data stacks. All these earlier data integration solutions are trying to be full, comprehensive solution, end-to-end. They try to own 100% of your data value chain. And that works up to a point. In general, you start getting a solution internally that works for 60% of your use cases. Then you need to do something that is a little bit outside of how these solution were thought about, then you have to do a huge hack.

… Airbyte has a few key components. First are the connectors, and the protocol that powers these connectors: How do you build a connector, how do you package and how you exchange data between sources and destinations, and all the tooling that goes into this.  Secondly, we have dashboards and API’s to control and orchestrate all these data pipelines. You have different ways to deploy: on a single node or you can deploy it on Kubernetes. Finally, people have written guides on how to actually put things like login, security, access control on top open source Airbyte.

Download a complete transcript of this episode by filling out the form below:

Highlights in the video version:

Related content:

Subscribe to our Newsletter:
We also publish a popular newsletter where we share highlights from recent episodes, trends in AI / machine learning / data, and a collection of recommendations.

[Photo by geralt on Pixabay.]