Three levers for making a model smaller
Given the hard constraints from the last lesson, the practical question becomes: how do you take a model that was trained without those constraints in mind and make it fit anyway? Three techniques, often used together, form the standard toolkit. Quantization reduces the numeric precision used to store a model's weights, for example converting 32-bit floating point numbers down to 8-bit integers. Since a weight's exact value rarely needs 32 bits of precision to still contribute usefully to a prediction, this typically shrinks the model to roughly a quarter of its original size and speeds up the arithmetic, at the cost of a small, usually acceptable, drop in accuracy.
Pruning takes a different angle: instead of representing every weight less precisely, it removes some weights and connections entirely. Not every parameter in a trained network contributes equally to its output, and pruning identifies the ones contributing least and cuts them out, producing a smaller, sparser network that does less computation per prediction. Knowledge distillation takes a third approach altogether, training a new, deliberately smaller 'student' model to mimic the behavior of a larger, more accurate 'teacher' model, rather than shrinking the teacher's own weights. The student learns to approximate the teacher's outputs, often achieving much of the teacher's accuracy in a fraction of the size.