A model that works isn't a model you understand
You've already seen how a network learns in Neural Networks & Deep Learning: gradient descent nudges millions of weights until the model's outputs get closer to what you want. That process tells you the model got better at a task, but it tells you almost nothing about how it's doing the task internally. Two networks can produce identical outputs on every input you've tried while relying on completely different internal strategies, and you'd have no way to tell them apart from the outside.
That gap matters more as models get deployed into higher-stakes settings. A model that gives the right answer for the wrong reason, say by picking up on a spurious pattern in the training data rather than genuinely solving the task, will often fail unpredictably the moment it hits a case where that spurious pattern doesn't hold. External evaluation, the kind covered in Model Evaluation & Metrics, can catch that a model is failing. It generally can't tell you why, and it can't tell you what the model would do in situations you didn't think to test.