Soft Targets

Overview: Knowledge Distillation

We finally do the episode we kept promising — a proper ground-up explanation of knowledge distillation: what it is, how the teacher-student mechanism actually works, why soft targets carry more signal than hard labels, and where this shows up in real systems being built right now.