Moderate Embeddings Question 74 of 223

How do you choose embedding dimensionality?

GenAI / LLM · Speak this in 60–90 seconds · Faridabad & Delhi NCR

PICTURE THIS: AN LLM TURN

Text inTokens

TransformerAttention

Text outNext token

Simple meaning

Higher dimensions can hold more nuance but cost more storage, RAM, and distance compute.

WHY — Tokens instead of words?

LLMs use tokens (not full words) because it helps them:

This is a process

question about Embeddings.

Panels listen for order,

trade-offs, and what you would actually do on a GenAI / LLM project - not buzzwords.

Stable token IDs

Each piece maps to a number the network can learn.

Fits the model

Fixed pieces are what transformers expect as input.

STEPS — What happens with tokens?

Before the model can read a sentence, it goes through these steps:

1
Higher dimensions can hold
more nuance but cost more storage, RAM, and distance compute.
2
Many teams start with
the model's native size, then consider Matryoshka truncation if supported.
3
Measure recall@k on your
corpus, not just public MTEB numbers.
4
Context mix
Attention looks at nearby tokens together.
5
Next token
The model scores what should come next.
6
Decode
IDs turn back into readable text.

EXAMPLE — See it in action

Let's see how a real sentence is tokenized (tokens may vary by model):

Input text

“Many teams start with the model's native size, then consider Matryoshka truncati”

Tokenized output

Manyteamsstartwiththemodel's

Token IDs (example)

2987408337471632900

Note: Actual tokens and IDs depend on the tokenizer (e.g., GPT, Llama, etc.).

Key takeaway

Higher dimensions can hold more nuance but cost more storage, RAM, and distance compute. Many teams start with the model's native size, then consider Matryoshka truncation if supported.