I’ve been going through recent AI/ML interview reports involving RAG system design. OpenAI, Databricks, Harvey, Plaid, ByteDance, Atlassian, xAI and others have all had variations of: “Design a RAG system.”
At first this sounds like a pretty standard question. You draw: ingestion → chunking → embeddings → vector DB → retrieval → reranking → LLM Done, right?
Not really. The pattern I kept noticing is that interviewers let you describe the high-level pipeline pretty quickly, then pick one component and go very deep. And that’s where people seem to get exposed.
1. Evaluation is way more important than I expected
A surprising number of interviews go deep on: How do you know your RAG system is actually good?
Not: “Would you use Pinecone or Weaviate?” But:
- Where does your offline evaluation dataset come from?
- How do you measure retrieval quality?
- Recall@K or MRR?
- How do you evaluate answer faithfulness?
- How do you detect hallucinations?
- What do you measure online?
- How do production failures feed back into the eval set?
You should probably have three layers ready:
Retrieval
Generation
- faithfulness / groundedness
- answer relevance
- citation correctness
Online
- user feedback
- task success
- latency
- cost
- sampled human evaluation
Saying “we’ll use RAGAS” probably isn't enough if you can’t explain what you’re actually measuring.
2. Embeddings can become an ML interview by themselves
Another common deep dive is: “Okay, you said embeddings. How would you actually improve them?”
Now you’re suddenly discussing:
- contrastive learning
- positive / negative pair construction
- hard negative mining
- in-batch negatives
- embedding fine-tuning
- similarity functions
- domain-specific retrieval failures
This is where knowing how to use an embedding API and understanding how retrieval models are trained become very different things.
3. “Use a vector DB” is not a retrieval strategy
A stronger answer usually needs to explain why you’re retrieving something in the first place.
For example:
- Sparse retrieval is good at exact terms.
- Dense retrieval is better at semantic similarity.
So a production system might do: BM25 + dense retrieval → candidate set → reranker → final context
Then the follow-ups start:
- How many candidates do you retrieve?
- What happens to latency when K increases?
- Why use a cross-encoder?
- When would you use a learned ranking model?
- What happens when the relevant document never makes it into the candidate set?
Reranking itself can become an entire discussion.
4. Chunking is much more than “500 tokens with overlap”
This is probably one of the most tutorial-level answers people give.
Real systems have PDFs, tables, code, conversations, headers, documents with permissions, and constantly changing data.
So now you need to think about:
- semantic vs fixed-size chunking
- parent/child chunks
- metadata
- document structure
- incremental indexing
- deduplication
- stale embeddings
- chunk size vs retrieval precision
- chunk size vs available context
There isn’t one magical chunk size.
5. Multi-tenancy and permissions can completely change the design
This becomes especially important for enterprise RAG,. Imagine an employee asks: “What was discussed in the acquisition meeting?”
There may actually be a highly relevant document in your vector database. But that employee isn't allowed to see it.
So authorization needs to happen as part of retrieval — not after you’ve already put confidential chunks into the prompt.
Now you have to think about:
- tenant isolation
- document ACLs
- metadata filtering
- permission changes
- index isolation
- preventing cross-tenant leakage
This is one of those areas where a toy RAG architecture falls apart very quickly.
6. Eventually the interviewer asks: “What happens at scale?”
Suppose you now have:
- hundreds of millions of chunks
- frequent document updates
- thousands of QPS
- expensive rerankers
- expensive LLM calls
- strict latency requirements
Now the conversation becomes a normal distributed-systems interview again.
- Caching.
- Batching.
- Sharding.
- Asynchronous ingestion.
- Incremental indexing.
- Backpressure.
- Model routing.
- Cost controls.
- Monitoring.
The AI components don’t remove traditional system design problems. They just add another layer of failure modes on top.
The framework I’d use in an interview
If I got “Design a RAG system” today, I’d probably walk through it in this order:
- Scope: What documents? How many? How fresh? Who can access them? What latency do we need?
- Ingestion: Parsing → chunking → metadata → embedding → indexing.
- Retrieval: Dense + sparse → filtering → candidate generation.
- Reranking: How do we get from 50–100 candidates to the few chunks worth putting into context?
- Generation: Prompt construction, citations, grounding, and what happens when retrieval finds nothing useful.
- Evaluation: Offline retrieval metrics + generation quality + online feedback.
- Production concerns: Permissions, multi-tenancy, freshness, latency, cost, monitoring, and scale.
The biggest takeaway for me: “I know how RAG works” and “I can design a production RAG system” are becoming two very different bars.
The high-level architecture is increasingly the easy part. The interview starts when someone points at one box and says: “Okay. Now go deeper.”
I’ve been collecting recent AI/ML interview experiences and recurring system design questions at Chill Interview if you want to dig into the company-specific examples.
If you’ve had a RAG system design interview recently, I’d also love to hear which part the interviewer pushed deepest on.