Moderate Prompting Question 140 of 223

Why does streaming matter in LLM APIs?

GenAI / LLM · Speak this in 60–90 seconds · Faridabad & Delhi NCR

PICTURE THIS: A SENTENCE BECOMES TOKENS

The model does not read letters like humans. It reads these pieces, then predicts the next one.

Simple meaning

Streaming sends tokens as they are generated instead of waiting for the full completion.

WHY — Prompting instead of guessing?

Why interviewers care about Prompting:

They are checking judgment

on Prompting.

A good answer names

the situation, the default choice, and one exception - that reads as experience.

Stay structured

Name the idea, why it exists, then one short example.

Close cleanly

End with when you use it and one common pitfall.

STEPS — What happens with tokens?

Before the model can read a sentence, it goes through these steps:

1
Streaming sends tokens as
they are generated instead of waiting for the full completion.
2
Perceived latency drops even
when total generation time is unchanged.
3
Your client must handle
partial JSON, cancellations, and incomplete tool calls.
4
Context mix
Attention looks at nearby tokens together.
5
Next token
The model scores what should come next.
6
Decode
IDs turn back into readable text.

EXAMPLE — See it in action

Here's a short line you can speak, broken into clear beats:

Say this line

“Perceived latency drops even when total generation time is unchanged.”

Break into beats

Perceivedlatencydropsevenwhentotal

Speaking order

2987408337471632900

Note: Adapt this scaffold to your own project — keep it under 60–90 seconds.

Key takeaway

Streaming sends tokens as they are generated instead of waiting for the full completion. Perceived latency drops even when total generation time is unchanged.