Project Lightspeed: Next-generation Spark Streaming

Karthik Ramasamy on faster and simpler tools for new streaming applications.

SubscribeApple • Android • Spotify • Stitcher • Google • AntennaPod • RSS.

Karthik Ramasamy, is the Head of Streaming at Databricks1. He has extensive experience in streaming, having led teams at Twitter (Apache Heron), Splunk, and Streamlio (Apache Pulsar).

#E4EEF7;">Subscribe to the Gradient Flow Newsletter.

We discussed some of the key areas addressed by Lightspeed (next-gen Spark Streaming) including:

  • Improving latency and ensuring that it’s predictable
  • Enhancing ecosystem for connectors
  • Simplifying deployment, operations, monitoring and troubleshooting
Figure 1: Key areas addressed by Project Lightspeed; Image from Karthik Ramasamy, used with permission.

Highlights in the video version:

Related content:

If you enjoyed this episode, please support our work by encouraging your friends and colleagues to subscribe to our newsletter:

[1] Ben Lorica is an advisor to Databricks and other startups.

[Image: Data Stream by Ben Lorica, generated with OpenAI DALL·E.]