Generative AI: Image, Video & Diffusion Models
From Words to Pixels: Text-to-Image Mechanics · 1/2

Teaching a model to connect words and images

TheThecatcatsatsatdowndown

A diffusion model on its own only knows how to turn noise into 'a plausible image,' it has no idea what you want that image to be. To steer generation with a text prompt, modern systems rely on a separate component trained specifically to align text and images in a shared representation space, the approach popularized by OpenAI's CLIP. CLIP-style models are trained on huge numbers of image-caption pairs scraped from the web, learning to place an image and its matching caption close together in a shared mathematical space while pushing mismatched pairs apart.

Once a model has learned that alignment, your text prompt can be converted into a representation that sits in the same space as image representations. During each denoising step, the diffusion process is nudged toward regions of that space consistent with your prompt's representation, which is what causes the emerging image to actually reflect the words you typed rather than being generic noise-to-image.