Turning text into pieces a model can use
Before any NLP system, rule-based, statistical, or neural, can do anything with text, it has to break that text into discrete units called tokens. This sounds trivial until you actually try to do it. Splitting on spaces seems obvious, but 'don't' isn't one word or two clean words, it's 'do' and 'n't' mashed together with an apostrophe, and a naive space-split would leave punctuation stuck to words, turning 'cat.' and 'cat' into two completely different tokens even though they mean the same thing. Casing adds another wrinkle: is 'The' the same token as 'the'? Most systems have to decide deliberately, not by accident.
Different languages make this even harder. English at least has spaces between most words, but Chinese, Japanese, and Thai text is written with no spaces between words at all, so a system has to figure out where one word ends and the next begins using context and learned patterns, not just whitespace. German famously builds enormous compound words by gluing smaller words together, which means a naive word-level tokenizer would treat every compound as a brand new, never-before-seen token. Tokenization is the unglamorous first step that every downstream NLP task, old or new, depends on getting right.