Nick Schrock on a new approach for managing, maintaining, and orchestrating data declaratively.
Subscribe: Apple • Android • Spotify • Stitcher • Google • AntennaPod • RSS.
Nick Schrock is founder and Elementl, the startup behind Dagster, a popular open source, data orchestration platform. We discussed recent trends in data engineering and infrastructure, and Dagster’s introduction of software-defined assets, a new approach to managing, maintaining, and orchestrating data declaratively.
Highlights in the video version:
- Introduction to Nick Schrock
Why focus on data orchestration?
Persona and use cases
Software testing for pipelines
Orchestration solution is central to ops
Declarative technology and Software-defined assets
Transition from classic texture to new software defined asset texture
Can I mix and match between different programming languages?
Where will you be at the end of the year?
Changing the way developers think and the modern data stack
DAGs
Fragmentation and consolidation
Open Source
Orchestration should handle different modalities
Toolsets and orchestration platform
Democratization, low code and no code data integration
Analysts and data
Related content:
- A video version of this conversation is available on our YouTube channel.
- Jeremiah Lowin: Dataflow Automation
- Haytham Abuelfutuh: Orchestrating Machine Learning Applications
- Data Pegacorns: Introducing the Data $100M Revenue Club
- The Data Integration Market
- Distributed Computing for AI: A Status Report
- Pardhu Gunnam and Mars Lan: Why You Need a Modern Metadata Platform
- Michel Tricot: Modernizing Data Integration
FREE report:
[Image by Ben Lorica.]