Easy Embeddings Question 8 of 223

Why are embeddings useful for search?

GenAI / LLM · Speak this in 60–90 seconds · Faridabad & Delhi NCR

QuestionEmbed query

SearchCompany docs

LLMAnswer with sources

Keyword search needs overlapping words.

LLMs use tokens (not full words) because it helps them:

They are checking judgment

on Embeddings.

A good answer names

the situation, the default choice, and one exception - that reads as experience.

Stable token IDs

Each piece maps to a number the network can learn.

Fits the model

Fixed pieces are what transformers expect as input.

Before the model can read a sentence, it goes through these steps:

1
Tokenization
Keyword search needs overlapping words.
2
Embeddings let you match
paraphrases and related ideas even when wording differs.
3
You embed the query
and documents, then return nearest neighbors in vector space.
4
Context mix
Attention looks at nearby tokens together.
5
Next token
The model scores what should come next.
6
Decode
IDs turn back into readable text.

Let's see how a real sentence is tokenized (tokens may vary by model):

Input text

“Embeddings let you match paraphrases and related ideas even when wording differs”

Tokenized output

Embeddingsletyoumatchparaphrasesand

Token IDs (example)

2987408337471632900

Note: Actual tokens and IDs depend on the tokenizer (e.g., GPT, Llama, etc.).

Keyword search needs overlapping words. Embeddings let you match paraphrases and related ideas even when wording differs.