Moderate Attention Question 82 of 223

Why is standard self-attention quadratic in sequence length?

GenAI / LLM · Speak this in 60–90 seconds · Faridabad & Delhi NCR

The model does not read letters like humans. It reads these pieces, then predicts the next one.

Every token attends to every other token, so scores scale as n times n.

Why interviewers care about Attention:

They are checking judgment

on Attention.

A good answer names

the situation, the default choice, and one exception - that reads as experience.

Stay structured

Name the idea, why it exists, then one short example.

Close cleanly

End with when you use it and one common pitfall.

Before the model can read a sentence, it goes through these steps:

1
Every token attends to
every other token, so scores scale as n times n.
2
Memory for the attention
matrix grows the same way.
3
That is why long
context is expensive and why kernel tricks and sparse attention exist.
4
Context mix
Attention looks at nearby tokens together.
5
Next token
The model scores what should come next.
6
Decode
IDs turn back into readable text.

Here's a short line you can speak, broken into clear beats:

Say this line

“Memory for the attention matrix grows the same way.”

Break into beats

Memoryfortheattentionmatrixgrows

Speaking order

2987408337471632900

Note: Adapt this scaffold to your own project — keep it under 60–90 seconds.

Every token attends to every other token, so scores scale as n times n. Memory for the attention matrix grows the same way.