High Attention Question 142 of 223

How do grouped-query and multi-query attention speed up inference?

GenAI / LLM · Speak this in 60–90 seconds · Faridabad & Delhi NCR

The model does not read letters like humans. It reads these pieces, then predicts the next one.

They share key and value heads across many query heads, shrinking the KV cache.

LLMs use tokens (not full words) because it helps them:

This is a process

question about Attention.

Panels listen for order,

trade-offs, and what you would actually do on a GenAI / LLM project - not buzzwords.

Stable token IDs

Each piece maps to a number the network can learn.

Fits the model

Fixed pieces are what transformers expect as input.

Before the model can read a sentence, it goes through these steps:

1
They share key and
value heads across many query heads, shrinking the KV cache.
2
Decode memory and bandwidth
drop, which raises tokens per second.
3
Quality can dip slightly
versus full multi-head attention, so labs pick GQA as a serving compromise.
4
Context mix
Attention looks at nearby tokens together.
5
Next token
The model scores what should come next.
6
Decode
IDs turn back into readable text.

Let's see how a real sentence is tokenized (tokens may vary by model):

Input text

“Decode memory and bandwidth drop, which raises tokens per second.”

Tokenized output

Decodememoryandbandwidthdropwhich

Token IDs (example)

2987408337471632900

Note: Actual tokens and IDs depend on the tokenizer (e.g., GPT, Llama, etc.).

They share key and value heads across many query heads, shrinking the KV cache. Decode memory and bandwidth drop, which raises tokens per second.