Watching a live model instead of trusting an old score
Monitoring a production model means continuously tracking two things: the distribution of its inputs and predictions, and its actual performance against real outcomes as they become available. Comparing today's live prediction distribution against the training distribution is often the earliest warning sign of drift, well before you have enough labeled outcomes to measure a real accuracy drop directly. Once true outcomes do come in, whether a loan defaulted, a customer churned, a transaction turned out fraudulent, that ground truth gets compared against what the model predicted, and that comparison is the real measure of how the model is doing right now, not the test-set score from months ago.
The practical setup is alerting, not manual dashboard-watching. A team sets a threshold for acceptable performance and drift, and an alert fires automatically when a live metric crosses it, the same way infrastructure teams alert on error rates instead of eyeballing logs all day. This turns model health into something the team responds to quickly, rather than something that surfaces only when a downstream business metric has already visibly suffered.