Why chunk at all?

You can't embed an entire book as a single vector. Cram too much text into one embedding and its meaning gets averaged into mush — a vector that's vaguely about everything and precisely about nothing, so retrieval gets worse. You'd also blow past the embedding model's input limit. The fix is chunking: splitting documents into smaller passages, each embedded and retrieved on its own.

The size trade-off

Chunk size is a balancing act:

• Too small (a single sentence) — you retrieve a precise fragment but lose the surrounding context the model needs to understand it.
• Too large (a whole chapter) — you keep context but dilute the meaning and waste prompt space on irrelevant text.
• A good starting point: 200–500 tokens per chunk, then tune based on your content and results.

Practical tips that matter

• Overlap chunks slightly (say 50 tokens) so an idea split across a boundary isn't lost from both sides.
• Split on natural boundaries — paragraphs, headings, sections — not mid-sentence. Structure-aware splitting beats blind character counting.
• Keep metadata with each chunk — source, title, URL, date — so you can cite it and filter on it later.
• Match chunks to your questions. If users ask broad questions, lean larger; for precise lookups, lean smaller.

Why it's worth the effort

Chunking is quietly one of the highest-leverage knobs in the whole pipeline. Bad chunks mean the right answer is never even retrievable, no matter how good your model is. A surprising number of "the RAG isn't working" problems are really chunking problems.

Garbage chunks in, garbage answers out. If retrieval is disappointing, look at how you split the documents before you blame the model or the database.

Try this: Take one of your own documents and split it two ways — by fixed character count, and by paragraph/heading. Read a few chunks from each. The structure-aware version will almost always produce passages that stand on their own, which is exactly what retrieval needs.