Unlocking the Power of Unstructured Data

Chang She on the power of Lance, a columnar data format for AI/ML.

Subscribe: Apple • Spotify • Overcast • Pocket Casts • AntennaPod • Podcast Addict • Amazon •  RSS.

Chang She is CEO and co-founder of LanceDB, an open-source database designed for multimodal AI applications, offering scalable vector search, streaming training data, and interactive exploration of large AI datasets. In this episode we discuss Lance, an open-source columnar data format that tackles the unique challenges posed by modern AI and machine learning workloads. Specifically engineered for efficiency, Lance addresses limitations of existing formats like Parquet and ORC by optimizing the storage and retrieval of large, complex data types, including images, videos, and vector embeddings.

Subscribe to the Gradient Flow Newsletter

Interview highlights – key sections from the video version:

1. Introduction to Lance and the Challenge of Unstructured Data
2. Overcoming Limitations of Existing Formats (Parquet, ORC)
3. Lance: A New Data Format for AI Workloads
4. Efficient Metadata Handling and Wide Data Support in Lance
5. Integrated Vector Indexing for AI Applications
6. LanceDB: A Scalable Vector Database Built on Lance Format
7. Real-World Use Cases: Images, Videos, and Large-Scale Datasets
8. Lance as a “One-Stop Shop” for AI Data Lakes
9. Comparison to Meta’s Nimble: Similarities and Differences
10. Open Source Ecosystem and Community Contributions
11. Key Use Cases: Data Exploration, Training, and Vector Search
12. Addressing the Limitations of Traditional Vector Search Systems
13. Exploratory Data Analysis for Unstructured Data with Lance
14. Multimodal Embeddings and Vector Search
15. Feature Stores and Their Evolving Role in AI
16. Putting LanceDB’s Vector Search to the Test
17. Embedding Pipelines, Ecosystem Integrations, and Deployment
18. Open Source and Enterprise Offerings from LanceDB
19. The Future of Lance: New Encodings, Integrations, and Governance

 

Related content:

• A video version of this conversation is available on our YouTube channel.
• Previous episodes pertaining to vector search can be found HERE.
• Choosing the Right Vector Search System
• Is Your Data Strategy Ready for Generative AI?
• Semih Salihoglu → The Intersection of LLMs, Knowledge Graphs, and Query Generation
• Philip Rathle → Supercharging AI with Graphs
• Ajay Kulkarni and Mike Freedman → Postgres: The Swiss Army Knife of Databases

---

If you enjoyed this episode, please support our work by encouraging your friends and colleagues to subscribe to our newsletter: