Machine Learning Foundations
Overfitting: When a Model Memorizes Instead of Learns · 1/2

Memorizing the textbook instead of understanding the subject

Overfitting happens when a model learns the training data too well, including its noise and quirks, instead of learning the general pattern that would apply to new data. Imagine a student who memorizes the exact wording and answers of every practice question instead of understanding the underlying math. They'll ace a test made of those exact questions, but fall apart the moment the wording changes even slightly. An overfit model does the same thing: it performs almost perfectly on the data it was trained on, but performs poorly on new, unseen examples.

This tends to happen when a model is too complex relative to the amount and variety of training data available, like giving that student a photographic memory but only 10 practice problems to study from. With so few examples, memorizing becomes easier and more tempting than actually learning the underlying pattern, and the model has enough capacity to do exactly that.