How does chunk size affect retrieval quality?
PICTURE THIS: A SENTENCE BECOMES TOKENS
The model does not read letters like humans. It reads these pieces, then predicts the next one.
Simple meaning
Tiny chunks match precisely but lose surrounding meaning.
WHY — Tokens instead of words?
LLMs use tokens (not full words) because it helps them:
question about Chunking.
trade-offs, and what you would actually do on a GenAI / LLM project - not buzzwords.
Each piece maps to a number the network can learn.
Fixed pieces are what transformers expect as input.
STEPS — What happens with tokens?
Before the model can read a sentence, it goes through these steps:
- 1Tiny chunks match precisely
but lose surrounding meaning.
- 2Huge chunks match vaguely
and waste the context window.
- 3Sweep sizes on a
labeled retrieval set
- 4there is no universal
512-token rule.
- 5Next token
The model scores what should come next.
- 6Decode
IDs turn back into readable text.
EXAMPLE — See it in action
Let's see how a real sentence is tokenized (tokens may vary by model):
Note: Actual tokens and IDs depend on the tokenizer (e.g., GPT, Llama, etc.).
Key takeaway
Tiny chunks match precisely but lose surrounding meaning. Huge chunks match vaguely and waste the context window.