GenAI and LLM interview questions

A token is a chunk of text the model reads and writes, often a subword rather than a full word.

Tokenizers split rare words, punctuation, and code into several pieces and may merge common phrases into one piece.

Tokenization converts raw text into a sequence of integer IDs from a fixed vocabulary.

Byte Pair Encoding starts from characters or bytes and repeatedly merges the most frequent adjacent pairs into new vocabulary items.

Input tokens are the prompt, system message, history, and any retrieved context you send in.

An embedding is a dense numeric vector that represents text, an image, or another item in a continuous space.

Cosine similarity measures the angle between two vectors, not their raw length.

Keyword search needs overlapping words.

Semantic search ranks documents by meaning rather than exact keyword overlap.

A transformer is a neural architecture that uses attention to relate all positions in a sequence in parallel.

RNNs process tokens one after another, which is slow to train and weak on long-range links.

An encoder builds bidirectional representations of the full input.

Attention itself has no sense of order, so the model needs position signals.

Attention lets the model weight how much each token should influence another token.

Self-attention computes attention among tokens of the same sequence.

Each token is projected into query, key, and value vectors.

Multi-head attention runs several attention operations in parallel with different projections.

Causal attention blocks a token from seeing future tokens.

GPT stands for Generative Pre-trained Transformer.