How Large Language Models Work · 1/2
Tokens, not words
LLMs don't read words the way you do. They break text into small chunks called tokens, sometimes a whole word, sometimes just part of one. The model turns each token into a list of numbers, and does math on those numbers to figure out what comes next.
This is why AI models can sometimes struggle with things like counting letters in a word: they're not seeing letters at all, they're seeing tokens.