What CNNs get for free that ViTs don't
The genuine tradeoff between CNNs and ViTs comes down to inductive bias. A CNN's architecture assumes nearby pixels matter most, and that assumption is correct often enough that the network doesn't need to rediscover it from data, it gets a useful head start built into its structure. A ViT has no such assumption, self-attention treats all patches as equally reachable from the start, which is powerful, but it also means the model has to learn spatial structure, like the fact that neighboring patches are usually related, entirely from examples.
This is why ViTs generally need significantly more training data than CNNs to reach comparable performance from scratch. Trained on a modest dataset, a plain ViT often underperforms a well-designed CNN, not because self-attention is a worse mechanism, but because the ViT has more to learn and less structural help doing it.