What goes wrong if the embedding tokenizer differs from the generator tokenizer?
PICTURE THIS: A SENTENCE BECOMES TOKENS
The model does not read letters like humans. It reads these pieces, then predicts the next one.
Simple meaning
Retrieval still works because embedders have their own tokenizer, but chunk token budgets for the prompt must use the generator's tokenizer.
WHY — Tokens instead of words?
LLMs use tokens (not full words) because it helps them:
who only read docs from people who shipped.
and tied to GenAI / LLM work.
Each piece maps to a number the network can learn.
Fixed pieces are what transformers expect as input.
STEPS — What happens with tokens?
Before the model can read a sentence, it goes through these steps:
- 1Retrieval still works because
embedders have their own tokenizer, but chunk token budgets for the prompt must use the generator's tokenizer.
- 2A chunk that 'fits'
under the embedder can overflow the LLM context.
- 3Count prompt tokens with
the chat model's tokenizer after assembly.
- 4Context mix
Attention looks at nearby tokens together.
- 5Next token
The model scores what should come next.
- 6Decode
IDs turn back into readable text.
EXAMPLE — See it in action
Let's see how a real sentence is tokenized (tokens may vary by model):
Note: Actual tokens and IDs depend on the tokenizer (e.g., GPT, Llama, etc.).
Key takeaway
Retrieval still works because embedders have their own tokenizer, but chunk token budgets for the prompt must use the generator's tokenizer. A chunk that 'fits' under the embedder can overflow the LLM context.