Roy Schwartz on maximizing limited resources when building large language models.
Subscribe: Apple • Spotify • Stitcher • Google • AntennaPod • Podcast Addict • RSS.
Roy Schwartz is Professor of Natural Language Processing at The Hebrew University of Jerusalem. We discussed a recent survey paper (co-written by Roy) that presents a broad overview of existing methods to improve NLP efficiency through the lens of traditional NLP pipelines. Our conversation covered the following key areas:
- Data: Using fewer training instances or better utilizing those available can increase efficiency.
- Model Design
- Training: pre-training and fine tuning.
- Inference and compression
Highlights in the video version:
- Motivation for writing the survey paper “Efficient Methods for Natural Language Processing”
Efficiency via Data
Building domain specific NLP models
Efficiency via Model Design
Training: efficiency via pre-training and fine tuning
Tips for training NLP models
Efficiency and model inference
Green AI
Detoxifying models and biases in datasets
Related content:
- A video version of this conversation is available on our YouTube channel.
- Efficient Methods for Natural Language Processing: A Survey
- Barret Zoph and Liam Fedus: Efficient Scaling of Language Models
- Connor Leahy and Yoav Shoham: Large Language Models
- FREE Report: 2022 Trends in Data, Machine Learning, and AI
- Machine Learning Trends You Need To Know
- What is Graph Intelligence?
- Mark Chen of OpenAI: How DALL·E works
- Jack Clark: The 2022 AI Index
- Piotr Żelasko: The Unreasonable Effectiveness of Speech Data
- fastdup: Introducing a new free tool for curating image datasets at scale
If you enjoyed this episode, please support our work by encouraging your friends and colleagues to subscribe to our newsletter:
[Image: Cuypers Library – Inside the Rijksmuseum by Ben Lorica.]