Why do teams L2-normalize embeddings?
PICTURE THIS: GIT FLOW
Simple meaning
After unit normalization, inner product equals cosine similarity, which simplifies indexes and scores.
WHY — Tokens instead of words?
LLMs use tokens (not full words) because it helps them:
on Embeddings.
the situation, the default choice, and one exception - that reads as experience.
Each piece maps to a number the network can learn.
Fixed pieces are what transformers expect as input.
STEPS — What happens with tokens?
Before the model can read a sentence, it goes through these steps:
- 1After unit normalization, inner
product equals cosine similarity, which simplifies indexes and scores.
- 2It also reduces length
bias where longer chunks look closer just because their vectors are larger.
- 3Confirm the embedder was
trained with the same convention.
- 4Context mix
Attention looks at nearby tokens together.
- 5Next token
The model scores what should come next.
- 6Decode
IDs turn back into readable text.
EXAMPLE — See it in action
Let's see how a real sentence is tokenized (tokens may vary by model):
Note: Actual tokens and IDs depend on the tokenizer (e.g., GPT, Llama, etc.).
Key takeaway
After unit normalization, inner product equals cosine similarity, which simplifies indexes and scores. It also reduces length bias where longer chunks look closer just because their vectors are larger.