Moderate Attention Question 83 of 223

What is a KV cache and why does it speed up generation?

GenAI / LLM · Speak this in 60–90 seconds · Faridabad & Delhi NCR

The model does not read letters like humans. It reads these pieces, then predicts the next one.

During decoding, keys and values for past tokens are reused instead of recomputed.

LLMs use tokens (not full words) because it helps them:

Attention questions separate people

who only read docs from people who shipped.

Keep it short, concrete,

and tied to GenAI / LLM work.

Stable token IDs

Each piece maps to a number the network can learn.

Fits the model

Fixed pieces are what transformers expect as input.

Before the model can read a sentence, it goes through these steps:

1
During decoding, keys and
values for past tokens are reused instead of recomputed.
2
The cache grows with
context and generated length.
3
It is the main
reason the first token is slower than later tokens, and why long chats consume GPU memory.
4
Context mix
Attention looks at nearby tokens together.
5
Next token
The model scores what should come next.
6
Decode
IDs turn back into readable text.

Let's see how a real sentence is tokenized (tokens may vary by model):

Input text

“The cache grows with context and generated length.”

Tokenized output

Thecachegrowswithcontextand

Token IDs (example)

2987408337471632900

Note: Actual tokens and IDs depend on the tokenizer (e.g., GPT, Llama, etc.).

During decoding, keys and values for past tokens are reused instead of recomputed. The cache grows with context and generated length.