Project Lightspeed: Next-generation Spark Streaming

Karthik Ramasamy on faster and simpler tools for new streaming applications.

SubscribeApple • Android • Spotify • Stitcher • Google • AntennaPod • RSS.

Karthik Ramasamy, is the Head of Streaming at Databricks1. He has extensive experience in streaming, having led teams at Twitter (Apache Heron), Splunk, and Streamlio (Apache Pulsar).

Subscribe to the Gradient Flow Newsletter.

We discussed some of the key areas addressed by Lightspeed (next-gen Spark Streaming) including:

  • Improving latency and ensuring that it’s predictable
  • Enhancing ecosystem for connectors
  • Simplifying deployment, operations, monitoring and troubleshooting
Figure 1: Key areas addressed by Project Lightspeed; Image from Karthik Ramasamy, used with permission.

Highlights in the video version:

Related content:

If you enjoyed this episode, please support our work by encouraging your friends and colleagues to subscribe to our newsletter:

[1] Ben Lorica is an advisor to Databricks and other startups.

[Image: Data Stream by Ben Lorica, generated with OpenAI DALL·E.]