Training on matched and mismatched pairs
CLIP, short for Contrastive Language-Image Pretraining, is the best-known example of how a shared embedding space actually gets built. The training data is millions of image-caption pairs scraped from the web, a photo alongside text that describes it. For each batch of these pairs, the model computes an embedding for every image and every caption, then compares every image embedding against every caption embedding in that batch.
The training objective, contrastive learning, pulls the embeddings of a correct image-caption pair together while pushing the embeddings of every mismatched pair in that batch apart. Concretely, if a batch has 256 image-caption pairs, the model wants the embedding of image 5 to be closest to the embedding of caption 5 out of all 256 captions in the batch, and increasingly dissimilar to the other 255. Repeated over millions of pairs, this forces the two encoders to converge on a shared geometry.