What is the difference between batch and online inference?
PICTURE THIS: RAG CHATBOT
QuestionEmbed query
SearchCompany docs
LLMAnswer with sources
Simple meaning
Batch scores many rows on a schedule
WHY — Serving instead of guessing?
Why interviewers care about Serving:
They want a clean
contrast on Serving, not two memorised paragraphs.
Say what changes for
the developer, then one case where picking wrong hurts.
Stay structured
Name the idea, why it exists, then one short example.
Close cleanly
End with when you use it and one common pitfall.
STEPS — What happens step by step?
Before you speak the answer, walk the interviewer through these steps:
- 1Batch scores many rows
on a schedule
- 2online answers one request
with low latency.
- 3How it works
Feature freshness differs.
- 4I pick based on
product UX.
- 5Common mistake
What juniors usually get wrong.
- 6Close
When you pick this over the alternative.
EXAMPLE — See it in action
Here's a short line you can speak, broken into clear beats:
Say this line
“Feature freshness differs.”
Break into beats
Featurefreshnessdiffers
Speaking order
29874083374
Note: Adapt this scaffold to your own project — keep it under 60–90 seconds.
Key takeaway
Batch scores many rows on a schedule online answers one request with low latency.