Inference runtimes: built to run, not to train
The frameworks used to train a model are usually not the frameworks used to run it on a phone or embedded device. Training frameworks are built for flexibility: defining new architectures, computing gradients, and running on large, power-abundant clusters. Once a model is trained and compressed, it typically gets converted into a format used by a dedicated on-device inference runtime, a piece of software built specifically to execute an already-trained model as efficiently as possible on constrained hardware, with none of the machinery needed for training left inside it. These runtimes are deliberately narrow: they load a fixed set of weights and execute a fixed computation graph, trimmed of anything not needed just to produce predictions.
This conversion step matters because it's where a lot of the deployment-specific optimization happens: operations get fused together to reduce overhead, memory is laid out to match how the target chip actually accesses it, and the runtime picks the fastest available implementation of each operation for the specific hardware it detects at startup. A model that ran perfectly well inside its training framework can still fail to convert cleanly, or convert but run slowly, if it uses operations the target runtime doesn't support well. Part of designing for edge deployment is knowing this conversion step exists and picking architectures that survive it gracefully.