Knowledge Distillation KD
Training a small model against the outputs of a large or high-precision model - especially the class-probability "dark knowledge" - so the small model approaches the original accuracy with far fewer parameters and less compute. A common route to edge deployment and real-time inference for physics-AI models.
What it is
Knowledge distillation (KD) trains a small model against the outputs of a teacher model. The teacher is typically larger, more accurate, or slower to run; the student is far smaller and carries the production inference load.
The difference from ordinary supervised learning is the supervision signal: normal training uses the one-hot hard label of the ground truth, while distillation additionally uses the teacher's soft labels - the class-probability distributions. Those distributions encode similarity relationships between classes (the so-called "dark knowledge"), which is what lets the student learn more than the hard labels contain.
Key points
- The loss has two parts: a soft-label distillation term (usually KL divergence, or
cross-entropy on temperature-scaled probabilities) plus a hard-label term. Temperature T controls how much information the soft label carries; larger T flattens the distribution and exposes inter-class relationships more clearly.
- Intermediate-feature distillation often beats output distillation: matching the teacher's
intermediate representations works markedly better than aligning output distributions alone on physical field tasks, because the spatial structure of the field lives in the features.
- The teacher need not be a larger model: a common pattern in physics AI is to use a better-
converged, higher-accuracy surrogate as the teacher and distill it into a small network that satisfies real-time constraints.
- Distillation and quantization are orthogonal and complementary: distilling first shrinks the
model and improves the soft-label distribution, and quantizing afterwards usually loses less accuracy than doing either alone.
Common pitfalls / boundaries
- Distillation does not guarantee surpassing the teacher. The student's capacity caps how close
it can get; expecting a small student to exceed a stronger teacher is unrealistic unless extra information is introduced.
- Temperature and loss weights need tuning. Too high a temperature flattens the distribution and
loses discriminability; dropping the hard-label term entirely lets the student learn the teacher's mistakes on samples the teacher gets wrong.
- Soft labels need redesigning for physical-quantity tasks. In classification the meaning of
class probabilities is obvious, but for field regression the definition of "dark knowledge" must be constructed deliberately (for example, the distribution of differences across multi-condition solutions).
- The distilled model still has to be exported and quantized. No step of the deployment chain
can be skipped.