Distillation¶
Training a small "student" model to mimic a large "teacher" model — same behavior, lower cost.
What is distillation¶
To fill.
Teacher-student setup¶
To fill.
Hard vs soft labels¶
To fill.
When distillation makes sense¶
To fill.
Trade-offs¶
To fill.
References¶
To fill.