Moderate GPT vs BERT Question 85 of 223

How does masked language modeling differ from next-token prediction?

GenAI / LLM · Speak this in 60–90 seconds · Faridabad & Delhi NCR

The model does not read letters like humans. It reads these pieces, then predicts the next one.

Masked LM hides random tokens and reconstructs them from the full sentence.

LLMs use tokens (not full words) because it helps them:

This is a process

question about GPT vs BERT.

Panels listen for order,

trade-offs, and what you would actually do on a GenAI / LLM project - not buzzwords.

Stable token IDs

Each piece maps to a number the network can learn.

Fits the model

Fixed pieces are what transformers expect as input.

Before the model can read a sentence, it goes through these steps:

1
Masked LM hides random
tokens and reconstructs them from the full sentence.
2
Next-token prediction always forecasts
the future from the past.
3
Embeddings
MLM is a denoising task
4
NTP is a generative
language model objective.
5
Next token
The model scores what should come next.
6
Decode
IDs turn back into readable text.

Let's see how a real sentence is tokenized (tokens may vary by model):

Input text

“Next-token prediction always forecasts the future from the past.”

Tokenized output

Nexttokenpredictionalwaysforecaststhe

Token IDs (example)

2987408337471632900

Note: Actual tokens and IDs depend on the tokenizer (e.g., GPT, Llama, etc.).

Masked LM hides random tokens and reconstructs them from the full sentence. Next-token prediction always forecasts the future from the past.