Pretrained ViTs and the rise of vision-language systems
In practice, most teams working with ViTs today aren't training one from scratch, they're starting from a model already pretrained on a massive, diverse image dataset and fine-tuning it on their specific task with a much smaller labeled dataset. This pretrain-then-fine-tune pattern sidesteps the data-hunger problem, the heavy lifting of learning general visual structure already happened during pretraining, and fine-tuning only needs to adapt that knowledge to a narrower task.
This is also why ViTs have become central to multimodal systems that combine vision and language, since a transformer-based image encoder produces patch-token representations that sit naturally alongside the token representations a language model already uses. That shared token-based structure is part of why transformers have become the common architecture underneath many of today's cutting-edge vision and vision-language systems.