Karthik Ramasamy on faster and simpler tools for new streaming applications.
Subscribe: Apple • Android • Spotify • Stitcher • Google • AntennaPod • RSS.
Karthik Ramasamy, is the Head of Streaming at Databricks1. He has extensive experience in streaming, having led teams at Twitter (Apache Heron), Splunk, and Streamlio (Apache Pulsar).
Subscribe to the Gradient Flow Newsletter.
We discussed some of the key areas addressed by Lightspeed (next-gen Spark Streaming) including:
- Improving latency and ensuring that it’s predictable
- Enhancing ecosystem for connectors
- Simplifying deployment, operations, monitoring and troubleshooting
Highlights in the video version:
- Introduction to Karthik Ramasamy
Next Generation Spark Streaming: Project Lightspeed
Latency in streaming systems
ML inference and making latency more predictable
Spark Streaming usage by language ; new operators/APIs
Relationship between stream processing engine and time series databases
Connectors and simplifying tasks for Project Lightspeed
Project Lightspeed user expectations and Spark Streaming Community
Is there anything on the academic side of streaming that has intrigued you?
Is streaming an area where people have tried to build specialized hardware?
Related content:
- A video version of this conversation is available on our YouTube channel.
- Summer of Orchestration
- The Data Integration Market
- The Data Pegacorns
- The Business Intelligence Index
- Ratio of Data Scientists to Data Engineers
If you enjoyed this episode, please support our work by encouraging your friends and colleagues to subscribe to our newsletter:
[1] Ben Lorica is an advisor to Databricks and other startups.
[Image: Data Stream by Ben Lorica, generated with OpenAI DALL·E.]