When should you cache embeddings versus cache completions?
PICTURE THIS: RAG CHATBOT
Simple meaning
Cache embeddings for documents and repeated queries
WHY — Cost instead of guessing?
Why interviewers care about Cost:
contrast on Cost, not two memorised paragraphs.
the developer, then one case where picking wrong hurts.
Name the idea, why it exists, then one short example.
End with when you use it and one common pitfall.
STEPS — What happens with tokens?
Before the model can read a sentence, it goes through these steps:
- 1Cache embeddings for documents
and repeated queries
- 2they are deterministic for
a model version.
- 3Cache completions only for
identical prompts and when stale answers are acceptable.
- 4Completion caches can serve
a wrong cached hallucination at scale.
- 5Next token
The model scores what should come next.
- 6Decode
IDs turn back into readable text.
EXAMPLE — See it in action
Here's a short line you can speak, broken into clear beats:
Note: Adapt this scaffold to your own project — keep it under 60–90 seconds.
Key takeaway
Cache embeddings for documents and repeated queries they are deterministic for a model version.