One scale for a whole tensor, or one scale per channel
The simplest way to quantize a weight tensor is per-tensor: compute a single scale and zero-point from the min and max values across the entire tensor, and use that one mapping for every value in it. This is cheap to compute and cheap to execute, since the same scale applies everywhere. The problem is that real weight tensors, especially convolutional filters, often have very different value ranges across different output channels. If one channel has weights an order of magnitude larger than another, a single shared scale forces a brutal tradeoff: either the scale is coarse enough to cover the large channel, wasting precision on the small one, or it's fine enough for the small channel and clips the large one.
Per-channel quantization fixes this by computing an independent scale (and often an independent zero-point) for each output channel of a weight tensor. Each channel gets a mapping tuned to its own range, so no channel is forced to share precision with a channel that behaves completely differently. The cost is more scale factors to store and a very slightly more complex dequantization step, but on real convolutional and linear layers this is a small price for a reliably better accuracy than per-tensor quantization at the same bit-width, which is why per-channel weight quantization is the default in most production quantization toolchains.