Integrating Fine-tuning and Preference Alignment in a Single Streamlined Process

Jiwoo Hong and Noah Lee on Streamlining Language Model Training with Odds Ratio Preference Optimization.

Subscribe: Apple • Spotify • Overcast • Pocket Casts • AntennaPod • Podcast Addict • Amazon •  RSS.

Jiwoo Hong and  Noah Lee of KAIST AI are co-authors of ORPO: Monolithic Preference Optimization without Reference Model. ORPO utilizes the odds ratio to learn preferences during fine-tuning, requiring significantly smaller datasets compared to traditional methods like RLHF and DPO. The method has garnered interest from the research community and industry players due to its efficiency, scalability, and potential to mitigate bias in language models.

Subscribe to the Gradient Flow Newsletter

Interview highlights – key sections from the video version:

1. ORPO (Odds Ratio Preference Optimization) and how it combines supervised fine-tuning and preference alignment
2. The Odds Ratio
3. ORPO’s Objective Function and Dataset Size
4. Dataset Size Comparison with RLHF
5. ORPO’s Scalability and Model Size
6. Data Requirements for Specific Tasks
7. Comparison with Other Methods
8. Single Dataset Approach and Preference Alignment
9. The Nature of the ORPO Dataset
10. ORPO’s Performance Compared to Traditional Methods
11. ORPO’s Place in the AI Toolbox
12. Evidence of ORPO’s Effectiveness
13. ORPO and Bias Mitigation
14. Adaptability
15. Implementation of ORPO
16. Community and Industry Reaction to ORPO
17. Creating ORPO Datasets
18. ORPO’s Efficiency and Future Directions

 

Related content:

• A video version of this conversation is available on our YouTube channel.
• Notebook:  Fine-tune Llama 3 with ORPO
• From Supervised Fine-Tuning to Online Feedback
• Customizing LLMs: When to Choose LoRA or Full Fine-Tuning
• Ken Liu → Machine Unlearning: Techniques, Challenges, and Future Directions
• Scaling Dictionary Learning for Safer AI Models
• The Art of Forgetting: Demystifying Unlearning in AI Models
• Is Your Data Strategy Ready for Generative AI?
• Nestor Maslej → 2024 Artificial Intelligence Index

---

If you enjoyed this episode, please support our work by encouraging your friends and colleagues to subscribe to our newsletter: