Exploring Next
Exploring Next — Ep 181 w/ Justy & Cody — MIT's new fine-tuning method lets LLMs learn new skills without losing old ones
MIT researchers developed self-distillation fine-tuning (SDFT), a technique that lets large language models learn new skills without forgetting old ones. By using a model's own in-context learning abilities as both teacher and student, SDFT solves the catastrophic forgetting problem that forces companies to maintain separate models for each task.