When precision and recall lie about what matters
Offline evaluation for recommenders typically holds out some known interactions and checks whether the model would have ranked those held-out items highly, using metrics like precision and recall at some cutoff. These metrics are useful and necessary, but they can look strong while the system is quietly failing at its actual job. A model can score well on held-out accuracy while still creating a filter bubble, repeatedly showing a user narrow variations of what they've already engaged with, because predicting the next likely click is not the same objective as maximizing long-term satisfaction or discovery.
A related failure is popularity bias: since popular items have far more interaction data, models trained to predict interactions naturally get better at predicting engagement with already-popular items, and offline metrics reward that, since correctly predicting a popular item's high engagement is easy points. The result is a feedback loop where popular items get recommended more, generating more interactions, reinforcing their popularity further, while items that might genuinely suit a user but lack an interaction history stay buried, regardless of quality.