From aligned embeddings to real tasks
Once a system has a well-aligned shared space, and often a generative model layered on top of it, a range of practical capabilities become possible. Visual question answering lets a system answer a written question about the contents of an image, 'what color is the car in this photo' being resolved by reasoning jointly over the image's embedding and the question's embedding. Image captioning runs the same alignment in the other direction, generating descriptive text for an image rather than answering a specific question about it.
Text-to-image generation goes a step further, using a text embedding to guide a generative model that produces entirely new pixels, rather than retrieving or describing an existing image. And systems that accept a mix of image and text in a single prompt, letting you upload a photo and ask a follow-up question about it in the same conversation, are combining all of this: encoding the image into the shared space, encoding your text into the same space, and reasoning across both jointly rather than treating them as separate inputs handled one after another.