Moderate Tokens Question 72 of 223

Why might 'Hello' and 'hello' produce different token sequences?

GenAI / LLM · Speak this in 60–90 seconds · Faridabad & Delhi NCR

The model does not read letters like humans. It reads these pieces, then predicts the next one.

Many BPE vocabularies treat capitalization and leading spaces as part of the token.

LLMs use tokens (not full words) because it helps them:

They are checking judgment

on Tokens.

A good answer names

the situation, the default choice, and one exception - that reads as experience.

Stable token IDs

Each piece maps to a number the network can learn.

Fits the model

Fixed pieces are what transformers expect as input.

Before the model can read a sentence, it goes through these steps:

1
Many BPE vocabularies treat
capitalization and leading spaces as part of the token.
2
'Hello' can be one
ID while 'hello' is another.
3
That is why prompt
formatting and consistent casing change token counts and sometimes behavior.
4
Context mix
Attention looks at nearby tokens together.
5
Next token
The model scores what should come next.
6
Decode
IDs turn back into readable text.

Let's see how a real sentence is tokenized (tokens may vary by model):

Input text

“'Hello' can be one ID while 'hello' is another.”

Tokenized output

'Hello'canbeoneIDwhile

Token IDs (example)

2987408337471632900

Note: Actual tokens and IDs depend on the tokenizer (e.g., GPT, Llama, etc.).

Many BPE vocabularies treat capitalization and leading spaces as part of the token. 'Hello' can be one ID while 'hello' is another.