What is batch versus online inference?
PICTURE THIS: RAG CHATBOT
QuestionEmbed query
SearchCompany docs
LLMAnswer with sources
Simple meaning
Batch scores many rows on a schedule
WHY — Deployment instead of guessing?
Why interviewers care about Deployment:
They want a clean
contrast on Deployment, not two memorised paragraphs.
Say what changes for
the developer, then one case where picking wrong hurts.
Stay structured
Name the idea, why it exists, then one short example.
Close cleanly
End with when you use it and one common pitfall.
STEPS — What happens step by step?
Before you speak the answer, walk the interviewer through these steps:
- 1Batch scores many rows
on a schedule
- 2online answers single requests
with latency budgets.
- 3How it works
Architecture differs.
- 4Give an example
Cost profiles differ too.
- 5Common mistake
What juniors usually get wrong.
- 6Close
When you pick this over the alternative.
EXAMPLE — See it in action
Here's a short line you can speak, broken into clear beats:
Say this line
“Architecture differs.”
Break into beats
Architecturediffers
Speaking order
29874083
Note: Adapt this scaffold to your own project — keep it under 60–90 seconds.
Key takeaway
Batch scores many rows on a schedule online answers single requests with latency budgets.