GenAI and LLM interview questions

A fixed word vocabulary would map rares to unknown and explode in size.

Many BPE vocabularies treat capitalization and leading spaces as part of the token.

Use the same tokenizer the model uses, such as tiktoken for many OpenAI models.

Higher dimensions can hold more nuance but cost more storage, RAM, and distance compute.

Dense vectors are short learned embeddings where every dimension is used.

After unit normalization, inner product equals cosine similarity, which simplifies indexes and scores.

A bi-encoder embeds query and document separately so documents can be precomputed.

Residuals add the block input to its output so gradients flow through deep stacks.

After attention mixes information across tokens, a position-wise MLP transforms each token independently.

Encoder-decoder models such as T5 shine at transduction: translate, summarize, or map a full input to a structured output.

Self-attention uses queries, keys, and values from the same sequence.

Every token attends to every other token, so scores scale as n times n.

During decoding, keys and values for past tokens are reused instead of recomputed.

BERT sees tokens on both sides of a masked position, which helps disambiguation for NLU.

Masked LM hides random tokens and reconstructs them from the full sentence.

State the role, allowed tools, output schema, and what to do when information is missing.

Prompts are code.

ReAct interleaves reasoning traces with actions such as search or calculator calls.

You constrain the model to emit valid JSON or schema-shaped text.

The query may be rewritten, then hybrid search returns top chunks, optionally reranked.