Turning an image into a sentence
A transformer expects a sequence of tokens, like the words in a sentence, so applying one directly to raw pixels isn't practical, an image has far too many pixels to treat each one as an individual token. The Vision Transformer, or ViT, solves this by chopping the image into a grid of fixed-size patches, commonly 16x16 pixels, flattening each patch into a single vector, and feeding that sequence of patch-tokens into a standard transformer encoder. A 224x224 image split into 16x16 patches becomes a sequence of 196 tokens, and from that point on, the transformer treats the image exactly like it would treat a sentence, applying the same self-attention mechanism across the sequence of patches.
Because a transformer has no inherent sense of order or position, position embeddings are added to each patch token so the model knows where in the image each patch came from. Without this step, the model would see the same set of patches whether they came from a normal photo or the same patches shuffled randomly, since self-attention itself treats the input as an unordered set.