Chunking: the decision everyone underestimates
Before anything gets embedded, documents have to be split into chunks, and this step quietly determines the ceiling on your system's quality. Chunk too large, say a whole 10-page PDF, and the embedding becomes a vague average of many different ideas, hurting retrieval precision. Chunk too small, say a single sentence, and you lose surrounding context that the LLM needs to answer correctly. Most production systems chunk by a few hundred tokens with some overlap between consecutive chunks, so an idea that spans a chunk boundary isn't cut in half.
Smarter chunking respects document structure: splitting on headings, paragraphs, or code function boundaries rather than a fixed character count. A markdown doc split along its own section headers usually retrieves far better than one sliced every 500 characters regardless of what's on the page.