Why can the number of tokens differ from the number of words?
PICTURE THIS: GIT FLOW
Simple meaning
Tokenizers split rare words, punctuation, and code into several pieces and may merge common phrases into one piece.
WHY — Tokens instead of words?
LLMs use tokens (not full words) because it helps them:
on Tokens.
the situation, the default choice, and one exception - that reads as experience.
Each piece maps to a number the network can learn.
Fixed pieces are what transformers expect as input.
STEPS — What happens with tokens?
Before the model can read a sentence, it goes through these steps:
- 1Tokenizers split rare words,
punctuation, and code into several pieces and may merge common phrases into one piece.
- 2Spaces, numbers, and non-English
scripts often use extra tokens.
- 3That is why a
100-word prompt can be far more or fewer than 100 tokens.
- 4Context mix
Attention looks at nearby tokens together.
- 5Next token
The model scores what should come next.
- 6Decode
IDs turn back into readable text.
EXAMPLE — See it in action
Let's see how a real sentence is tokenized (tokens may vary by model):
Note: Actual tokens and IDs depend on the tokenizer (e.g., GPT, Llama, etc.).
Key takeaway
Tokenizers split rare words, punctuation, and code into several pieces and may merge common phrases into one piece. Spaces, numbers, and non-English scripts often use extra tokens.