Why soft labels carry more information than hard labels
In ordinary training, a model learns from hard labels: an image of a cat is labeled 'cat' with 100% confidence and everything else with 0%. Knowledge distillation instead trains a smaller student model to match the full output distribution of a larger, already-trained teacher model, its soft labels. If the teacher, looking at a photo of a wolf, assigns 85% probability to 'wolf' but also 10% to 'husky' and 3% to 'fox', that spread is genuinely informative: it tells the student that wolves and huskies share visually meaningful features, in a way a hard label of just 'wolf' never could. The student is trained to reproduce this whole distribution, not just to get the top answer right.
This matters because the relative probabilities the teacher assigns to wrong answers encode a kind of similarity structure the teacher learned from data, effectively summarizing what the teacher knows about how classes relate to each other. A student trained only on hard labels has to rediscover that structure from raw data alone. A student trained on soft labels gets a considerable amount of that structure handed to it directly, which is a large part of why distilled students often reach an accuracy that plain training on hard labels at the same model size would not.